2024-09-18T14:10:33.512 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T14:10:33.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T14:10:33.616 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/pool-create-delete} email: skanta@redhat.com first_in_suite: false job_id: '7911101' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T14:10:33.617 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T14:10:33.617 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T14:10:33.618 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T14:10:33.619 INFO:teuthology.task.internal:Checking packages... 2024-09-18T14:10:33.643 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T14:10:33.643 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T14:10:33.643 INFO:teuthology.packaging:ref: None 2024-09-18T14:10:33.643 INFO:teuthology.packaging:tag: None 2024-09-18T14:10:33.643 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T14:10:33.643 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:10:33.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T14:10:33.785 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:10:33.786 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T14:10:33.792 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T14:10:33.792 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T14:10:33.826 INFO:teuthology.task.internal:Saving configuration 2024-09-18T14:10:33.841 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T14:10:33.905 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T14:10:33.932 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 14:04:35.546353', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T14:10:33.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 14:04:35.547438', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T14:10:33.955 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T14:10:33.968 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-18T14:10:33.968 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-18T14:10:33.968 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T14:10:34.068 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5666c0bac0>, signals=[15]) 2024-09-18T14:10:34.068 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T14:10:34.075 INFO:teuthology.task.internal:Opening connections... 2024-09-18T14:10:34.075 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:10:34.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:10:34.168 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:10:34.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:10:34.265 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T14:10:34.270 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-09-18T14:10:34.277 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-09-18T14:10:34.277 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-09-18T14:10:34.323 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="20.04" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=focal 2024-09-18T14:10:34.324 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=focal 2024-09-18T14:10:34.325 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-09-18T14:10:34.350 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-09-18T14:10:34.356 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-09-18T14:10:34.357 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-09-18T14:10:34.405 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="Ubuntu" 2024-09-18T14:10:34.405 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T14:10:34.405 INFO:teuthology.orchestra.run.smithi174.stdout:ID=ubuntu 2024-09-18T14:10:34.405 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE=debian 2024-09-18T14:10:34.405 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T14:10:34.405 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="20.04" 2024-09-18T14:10:34.405 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T14:10:34.406 INFO:teuthology.orchestra.run.smithi174.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T14:10:34.406 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T14:10:34.406 INFO:teuthology.orchestra.run.smithi174.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T14:10:34.406 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_CODENAME=focal 2024-09-18T14:10:34.406 INFO:teuthology.orchestra.run.smithi174.stdout:UBUNTU_CODENAME=focal 2024-09-18T14:10:34.406 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-09-18T14:10:34.424 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T14:10:34.432 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T14:10:34.495 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T14:10:34.504 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-09-18T14:10:34.506 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-09-18T14:10:34.512 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T14:10:34.544 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T14:10:34.544 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-09-18T14:10:34.555 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-09-18T14:10:34.563 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T14:10:34.637 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T14:10:34.670 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T14:10:34.670 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T14:10:34.670 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T14:10:34.671 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T14:10:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:10:34.676 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-18T14:10:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:10:34.677 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-18T14:10:34.683 INFO:teuthology.orchestra.run.smithi104.stdout:5.4.0-124-generic 2024-09-18T14:10:34.684 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.4.0-124-generic 2024-09-18T14:10:34.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-09-18T14:10:34.685 INFO:teuthology.orchestra.run.smithi174.stdout:5.4.0-124-generic 2024-09-18T14:10:34.685 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.4.0-124-generic 2024-09-18T14:10:34.685 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-09-18T14:10:34.771 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-18T14:10:34.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-09-18T14:10:34.966 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T14:10:35.059 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:10:35.059 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T14:10:35.081 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:10:35.114 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T14:10:35.163 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T14:10:35.251 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T14:10:35.343 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T14:10:35.387 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T14:10:35.390 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T14:10:35.390 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T14:10:35.411 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T14:10:35.411 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T14:10:35.415 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T14:10:35.415 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T14:10:35.421 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T14:10:35.425 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T14:10:35.427 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T14:10:35.427 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T14:10:35.427 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T14:10:35.439 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T14:10:35.439 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T14:10:35.468 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T14:10:35.517 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T14:10:35.539 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T14:10:35.657 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,597 kB] 2024-09-18T14:10:35.679 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,597 kB] 2024-09-18T14:10:36.044 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T14:10:36.058 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T14:10:36.066 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T14:10:36.066 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T14:10:36.067 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T14:10:36.107 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T14:10:36.125 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T14:10:36.125 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T14:10:36.126 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T14:10:36.146 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T14:10:36.155 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T14:10:36.156 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T14:10:36.163 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T14:10:36.163 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T14:10:36.164 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T14:10:36.173 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T14:10:36.173 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T14:10:36.173 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T14:10:36.208 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T14:10:36.224 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T14:10:36.225 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T14:10:36.228 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T14:10:36.229 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T14:10:36.229 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T14:10:36.237 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T14:10:36.237 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-18T14:10:36.259 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T14:10:36.266 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T14:10:36.273 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T14:10:36.273 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-18T14:10:36.295 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T14:10:36.304 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T14:10:36.304 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T14:10:36.305 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T14:10:36.305 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T14:10:36.305 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T14:10:36.310 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T14:10:36.317 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T14:10:36.321 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T14:10:36.340 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T14:10:36.345 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T14:10:36.346 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T14:10:36.348 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T14:10:36.348 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T14:10:36.386 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T14:10:36.387 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T14:10:36.388 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T14:10:36.389 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T14:10:36.389 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T14:10:36.398 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T14:10:36.409 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T14:10:36.409 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T14:10:36.410 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T14:10:36.411 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T14:10:36.491 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T14:10:36.491 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T14:10:36.492 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T14:10:39.850 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 22.0 MB in 3s (7,507 kB/s) 2024-09-18T14:10:40.305 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 22.0 MB in 4s (6,172 kB/s) 2024-09-18T14:10:41.327 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-18T14:10:41.356 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T14:10:41.447 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-18T14:10:41.662 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-09-18T14:10:41.663 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-09-18T14:10:41.858 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-09-18T14:10:41.859 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T14:10:41.859 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T14:10:41.859 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T14:10:41.860 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-09-18T14:10:41.860 INFO:teuthology.orchestra.run.smithi104.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T14:10:41.860 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-09-18T14:10:41.860 INFO:teuthology.orchestra.run.smithi104.stdout: thermald 2024-09-18T14:10:41.878 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T14:10:41.902 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T14:10:41.914 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-09-18T14:10:41.914 INFO:teuthology.orchestra.run.smithi104.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T14:10:41.914 INFO:teuthology.orchestra.run.smithi104.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T14:10:41.915 INFO:teuthology.orchestra.run.smithi104.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T14:10:41.916 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages will be upgraded: 2024-09-18T14:10:41.917 INFO:teuthology.orchestra.run.smithi104.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T14:10:41.981 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T14:10:42.117 INFO:teuthology.orchestra.run.smithi104.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T14:10:42.117 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 77.2 MB of archives. 2024-09-18T14:10:42.117 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T14:10:42.117 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T14:10:42.201 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-18T14:10:42.201 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-18T14:10:42.410 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-09-18T14:10:42.410 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T14:10:42.410 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T14:10:42.410 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T14:10:42.411 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-09-18T14:10:42.411 INFO:teuthology.orchestra.run.smithi174.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T14:10:42.411 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-09-18T14:10:42.411 INFO:teuthology.orchestra.run.smithi174.stdout: thermald 2024-09-18T14:10:42.466 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-09-18T14:10:42.466 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T14:10:42.467 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T14:10:42.467 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T14:10:42.468 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2024-09-18T14:10:42.469 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T14:10:42.525 INFO:teuthology.orchestra.run.smithi174.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T14:10:42.525 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 77.2 MB of archives. 2024-09-18T14:10:42.525 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T14:10:42.525 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T14:10:42.831 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T14:10:42.926 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T14:10:43.265 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T14:10:43.377 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T14:10:43.377 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T14:10:43.377 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T14:10:43.468 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T14:10:43.478 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T14:10:43.582 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T14:10:44.221 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 77.2 MB in 1s (72.6 MB/s) 2024-09-18T14:10:44.435 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T14:10:44.882 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T14:10:44.882 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T14:10:44.882 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T14:10:45.209 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T14:10:45.228 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T14:10:45.921 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 77.2 MB in 3s (23.4 MB/s) 2024-09-18T14:10:46.091 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T14:10:46.096 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:10:46.100 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T14:10:46.181 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:10:47.976 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T14:10:47.981 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:10:48.039 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T14:10:48.052 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:10:48.173 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:10:48.188 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:10:49.250 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T14:10:49.264 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:10:49.308 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:10:50.115 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T14:10:50.130 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:10:50.220 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:10:51.285 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T14:10:51.300 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:10:51.342 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:10:54.318 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:10:54.438 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:10:54.789 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:10:54.941 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:10:55.243 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T14:10:55.256 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T14:10:55.293 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T14:10:56.131 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:10:56.255 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:10:56.581 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:10:56.700 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:10:57.062 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T14:10:57.076 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T14:10:57.119 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T14:11:00.599 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T14:11:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:11:00.657 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:02.213 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T14:11:02.234 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:11:02.271 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:03.393 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:11:03.514 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:11:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T14:11:04.127 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:04.253 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T14:11:04.379 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:04.978 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:11:05.101 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:11:05.417 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T14:11:05.547 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:05.580 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:05.673 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T14:11:05.798 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:06.752 INFO:teuthology.orchestra.run.smithi174.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:11:06.752 INFO:teuthology.orchestra.run.smithi174.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:06.891 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:06.992 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:08.076 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T14:11:08.201 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T14:11:08.328 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:08.401 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T14:11:08.402 INFO:teuthology.orchestra.run.smithi174.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:08.755 INFO:teuthology.orchestra.run.smithi104.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:11:08.756 INFO:teuthology.orchestra.run.smithi104.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:08.902 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:10.070 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T14:11:10.196 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T14:11:10.331 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:11:10.405 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T14:11:10.405 INFO:teuthology.orchestra.run.smithi104.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:30.624 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T14:11:30.624 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub' 2024-09-18T14:11:30.633 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T14:11:30.650 INFO:teuthology.orchestra.run.smithi174.stdout:Generating grub configuration file ... 2024-09-18T14:11:30.967 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:11:30.987 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:31.395 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T14:11:31.406 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T14:11:32.018 INFO:teuthology.orchestra.run.smithi104.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T14:11:32.018 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub' 2024-09-18T14:11:32.025 INFO:teuthology.orchestra.run.smithi104.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T14:11:32.044 INFO:teuthology.orchestra.run.smithi104.stdout:Generating grub configuration file ... 2024-09-18T14:11:32.354 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:11:32.375 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:32.742 INFO:teuthology.orchestra.run.smithi104.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T14:11:32.753 INFO:teuthology.orchestra.run.smithi104.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T14:11:33.462 INFO:teuthology.orchestra.run.smithi174.stdout:done 2024-09-18T14:11:33.891 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-09-18T14:11:33.932 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-09-18T14:11:33.932 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-09-18T14:11:33.932 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-09-18T14:11:33.932 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 20 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.4.0.196.194 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-09-18T14:11:33.933 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-09-18T14:11:33.934 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T14:11:33.934 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T14:11:33.934 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T14:11:33.934 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-18T14:11:33.934 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T14:11:33.934 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-09-18T14:11:34.010 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-18T14:11:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:11:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T14:11:34.148 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T14:11:34.165 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T14:11:34.673 INFO:teuthology.orchestra.run.smithi104.stdout:done 2024-09-18T14:11:35.108 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-09-18T14:11:35.157 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-09-18T14:11:35.157 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-09-18T14:11:35.157 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-09-18T14:11:35.157 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-09-18T14:11:35.157 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 20 2024-09-18T14:11:35.157 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.4.0.196.194 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-09-18T14:11:35.158 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-09-18T14:11:35.158 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T14:11:35.158 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T14:11:35.159 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T14:11:35.159 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T14:11:35.159 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T14:11:35.159 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get clean 2024-09-18T14:11:35.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-09-18T14:11:35.380 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:11:35.380 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T14:11:35.385 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T14:11:35.403 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T14:11:36.009 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T14:11:36.036 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T14:11:36.106 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T14:11:36.302 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-18T14:11:36.302 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-18T14:11:36.533 INFO:teuthology.orchestra.run.smithi174.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T14:11:36.534 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T14:11:36.536 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 20 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.4.0.196.194 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T14:11:36.554 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-09-18T14:11:36.555 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-09-18T14:11:36.555 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-09-18T14:11:36.555 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-09-18T14:11:36.555 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-18T14:11:36.559 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.Jot0YlRKdZ 2024-09-18T14:11:36.559 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Jot0YlRKdZ 2024-09-18T14:11:36.574 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.Jot0YlRKdZ 2024-09-18T14:11:36.655 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.Jot0YlRKdZ is 9KB 2024-09-18T14:11:36.669 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.Jot0YlRKdZ 2024-09-18T14:11:36.675 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T14:11:36.729 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:11:36.729 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T14:11:36.729 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T14:11:36.766 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T14:11:36.766 DEBUG:teuthology.orchestra.run.smithi174:> sudo update-grub 2024-09-18T14:11:37.241 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-18T14:11:37.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T14:11:37.336 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-18T14:11:37.561 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-09-18T14:11:37.561 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-09-18T14:11:37.798 INFO:teuthology.orchestra.run.smithi104.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T14:11:37.799 INFO:teuthology.orchestra.run.smithi104.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T14:11:37.801 DEBUG:teuthology.orchestra.run.smithi104:> dpkg -s linux-image-generic 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Package: linux-image-generic 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Status: install ok installed 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Priority: optional 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Section: kernel 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Installed-Size: 20 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: amd64 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Source: linux-meta 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Version: 5.4.0.196.194 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T14:11:37.824 INFO:teuthology.orchestra.run.smithi104.stdout:Recommends: thermald 2024-09-18T14:11:37.825 INFO:teuthology.orchestra.run.smithi104.stdout:Description: Generic Linux kernel image 2024-09-18T14:11:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: This package will always depend on the latest generic kernel image 2024-09-18T14:11:37.825 INFO:teuthology.orchestra.run.smithi104.stdout: available. 2024-09-18T14:11:37.825 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-18T14:11:37.873 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.A1UZDvmsUY 2024-09-18T14:11:37.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub/grub.cfg /tmp/tmp.A1UZDvmsUY 2024-09-18T14:11:37.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.A1UZDvmsUY 2024-09-18T14:11:37.962 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.A1UZDvmsUY is 9KB 2024-09-18T14:11:37.978 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.A1UZDvmsUY 2024-09-18T14:11:37.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T14:11:38.001 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:11:38.001 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T14:11:38.001 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T14:11:38.042 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub' 2024-09-18T14:11:38.049 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T14:11:38.064 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-09-18T14:11:38.076 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T14:11:38.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo update-grub 2024-09-18T14:11:38.355 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:11:38.366 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:38.714 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T14:11:38.726 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T14:11:39.287 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub' 2024-09-18T14:11:39.298 INFO:teuthology.orchestra.run.smithi104.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T14:11:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-09-18T14:11:39.569 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:11:39.581 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:11:39.680 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-09-18T14:11:39.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-09-18T14:11:39.916 INFO:teuthology.orchestra.run.smithi104.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T14:11:39.927 INFO:teuthology.orchestra.run.smithi104.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T14:11:40.740 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-09-18T14:11:40.742 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-09-18T14:12:09.713 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-18T14:12:09.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T14:12:09.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:12:10.746 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T14:12:10.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-18T14:12:10.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:12:44.221 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-18T14:12:53.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T14:12:53.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:13:10.767 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T14:13:11.613 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-18T14:13:19.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-18T14:13:19.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:13:20.469 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-18T14:13:21.241 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-18T14:13:21.241 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T14:13:21.241 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-18T14:13:21.248 INFO:teuthology.orchestra.run.smithi104.stdout:5.4.0-196-generic 2024-09-18T14:13:21.248 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T14:13:21.248 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T14:13:21.248 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T14:13:22.250 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T14:13:22.250 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T14:13:22.341 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-09-18T14:13:22.376 DEBUG:teuthology.parallel:result is None 2024-09-18T14:13:23.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T14:13:23.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:13:23.861 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-18T14:13:24.443 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T14:13:24.443 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-196-generic"... 2024-09-18T14:13:24.443 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-18T14:13:24.449 INFO:teuthology.orchestra.run.smithi174.stdout:5.4.0-196-generic 2024-09-18T14:13:24.449 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T14:13:24.449 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T14:13:24.449 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T14:13:25.450 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-18T14:13:25.450 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T14:13:25.558 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-09-18T14:13:25.587 DEBUG:teuthology.parallel:result is None 2024-09-18T14:13:25.587 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T14:13:25.596 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T14:13:25.596 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T14:13:25.600 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T14:13:25.607 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T14:13:25.643 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T14:13:25.705 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T14:13:25.705 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T14:13:25.708 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T14:13:25.725 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T14:13:25.833 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T14:13:25.833 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T14:13:25.836 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T14:13:25.854 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:13:25.863 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:13:25.864 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:13:25.874 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:13:25.876 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T14:13:25.883 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T14:13:25.883 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T14:13:25.919 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T14:13:25.935 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T14:13:25.944 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T14:13:25.944 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T14:13:25.947 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T14:13:25.983 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T14:13:25.990 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T14:13:26.040 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:13:26.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T14:13:26.061 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T14:13:26.068 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T14:13:26.075 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:13:26.075 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T14:13:26.138 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-09-18T14:13:26.140 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-09-18T14:13:26.190 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T14:13:26.198 INFO:teuthology.task.internal:Starting timer... 2024-09-18T14:13:26.198 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T14:13:26.277 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T14:13:26.338 DEBUG:teuthology.task.selinux:Excluding smithi104: OS 'ubuntu' does not support SELinux 2024-09-18T14:13:26.338 DEBUG:teuthology.task.selinux:Excluding smithi174: OS 'ubuntu' does not support SELinux 2024-09-18T14:13:26.338 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T14:13:26.338 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T14:13:26.339 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T14:13:26.339 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T14:13:26.369 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-18T14:13:26.372 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-18T14:13:26.481 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T14:13:26.492 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T14:13:26.493 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T14:19:01.542 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-09-18T14:19:01.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-18T14:19:01.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:19:01.634 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-18T14:19:01.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-18T14:19:01.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T14:19:01.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:19:01.847 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-18T14:19:01.971 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T14:19:01.971 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T14:19:01.982 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T14:19:01.982 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T14:19:01.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T14:19:01.985 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T14:19:01.986 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T14:19:02.017 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T14:19:02.017 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Command line: ntpd -gq 2024-09-18T14:19:02.018 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: proto: precision = 0.086 usec (-23) 2024-09-18T14:19:02.018 INFO:teuthology.orchestra.run.smithi104.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:19:02.019 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:19:02.019 INFO:teuthology.orchestra.run.smithi104.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:19:02.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:19:02.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T14:19:02.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T14:19:02.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T14:19:02.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Listen normally on 3 enp3s0f1 172.21.15.104:123 2024-09-18T14:19:02.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Listen normally on 4 lo [::1]:123 2024-09-18T14:19:02.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce89%5]:123 2024-09-18T14:19:02.021 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:02 ntpd[13755]: Listening on routing socket on fd #22 for interface updates 2024-09-18T14:19:02.022 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T14:19:02.022 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Command line: ntpd -gq 2024-09-18T14:19:02.022 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: proto: precision = 0.116 usec (-23) 2024-09-18T14:19:02.023 INFO:teuthology.orchestra.run.smithi174.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:19:02.023 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:19:02.023 INFO:teuthology.orchestra.run.smithi174.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:19:02.024 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:19:02.024 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T14:19:02.024 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T14:19:02.024 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T14:19:02.024 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Listen normally on 3 enp3s0f1 172.21.15.174:123 2024-09-18T14:19:02.024 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Listen normally on 4 lo [::1]:123 2024-09-18T14:19:02.025 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:771b%5]:123 2024-09-18T14:19:02.025 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:02 ntpd[13719]: Listening on routing socket on fd #22 for interface updates 2024-09-18T14:19:03.018 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:03 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:03.018 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:03 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:03.018 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:03 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:03.018 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:03 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:03.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:03 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:03.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:03 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:03.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:03.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:03.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:03.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:03.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:03.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:03.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:03.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:03 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:04.018 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:04 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:04.018 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:04 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:04.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:04 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:04.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:04 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:04.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:04 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:04.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:04 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:04.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:04 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:04.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:04 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:05.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:05 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:05.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:05.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:05.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:05.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:05.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:05 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:06.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:06 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:06.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:06 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:06.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:06 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:06.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:06 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:06.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:06 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:06.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:06 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:07.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.021 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:07.021 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:07 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:07.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:07.024 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:07.024 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:07.024 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:07.024 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:07 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:08.020 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:08 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:08.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:08 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:08.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:08 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:08.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:08 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:08.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:08 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:08.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:08 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:09.019 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:09.020 INFO:teuthology.orchestra.run.smithi104.stdout:18 Sep 14:19:09 ntpd[13755]: ntpd: time slew +0.000916 s 2024-09-18T14:19:09.020 INFO:teuthology.orchestra.run.smithi104.stdout:ntpd: time slew +0.000916s 2024-09-18T14:19:09.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T14:19:09.020 INFO:teuthology.orchestra.run.smithi104.stderr:18 Sep 14:19:09 ntpd[13755]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T14:19:09.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:09.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:09.021 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:09.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:09.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:19:09.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:19:09.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:19:09.022 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:19:09.022 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 14:19:09 ntpd[13719]: ntpd: time slew +0.000512 s 2024-09-18T14:19:09.022 INFO:teuthology.orchestra.run.smithi174.stdout:ntpd: time slew +0.000512s 2024-09-18T14:19:09.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T14:19:09.023 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 14:19:09 ntpd[13719]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T14:19:09.080 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T14:19:09.081 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-09-18T14:19:09.081 INFO:teuthology.orchestra.run.smithi104.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.081 INFO:teuthology.orchestra.run.smithi104.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.081 INFO:teuthology.orchestra.run.smithi104.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.081 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.109 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T14:19:09.109 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-09-18T14:19:09.110 INFO:teuthology.orchestra.run.smithi174.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.110 INFO:teuthology.orchestra.run.smithi174.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.110 INFO:teuthology.orchestra.run.smithi174.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.110 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:19:09.110 INFO:teuthology.run_tasks:Running task install... 2024-09-18T14:19:09.118 DEBUG:teuthology.task.install:project ceph 2024-09-18T14:19:09.119 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T14:19:09.119 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T14:19:09.119 INFO:teuthology.task.install:Using flavor: default 2024-09-18T14:19:09.124 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T14:19:09.125 INFO:teuthology.task.install:extra packages: [] 2024-09-18T14:19:09.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-key list | grep Ceph 2024-09-18T14:19:09.126 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-key list | grep Ceph 2024-09-18T14:19:09.221 INFO:teuthology.orchestra.run.smithi104.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T14:19:09.255 INFO:teuthology.orchestra.run.smithi174.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T14:19:09.281 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T14:19:09.281 INFO:teuthology.orchestra.run.smithi104.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T14:19:09.283 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T14:19:09.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:19:09.319 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T14:19:09.319 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T14:19:09.319 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T14:19:09.320 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:19:09.427 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T14:19:09.427 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:19:09.460 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T14:19:09.460 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:19:09.539 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:19:09.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T14:19:09.557 DEBUG:teuthology.orchestra.run.smithi104:> sudo apt-get update 2024-09-18T14:19:09.572 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:19:09.572 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T14:19:09.588 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-18T14:19:09.731 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T14:19:09.783 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T14:19:09.794 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T14:19:09.805 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T14:19:09.816 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T14:19:09.828 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T14:19:09.839 INFO:teuthology.orchestra.run.smithi104.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T14:19:09.846 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:19:09.850 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T14:19:09.872 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T14:19:09.873 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:19:09.873 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T14:19:09.883 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T14:19:09.905 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T14:19:09.928 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T14:19:09.942 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T14:19:10.015 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T14:19:10.024 INFO:teuthology.orchestra.run.smithi104.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T14:19:10.383 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 28.4 kB in 1s (44.8 kB/s) 2024-09-18T14:19:10.523 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 28.4 kB in 1s (37.9 kB/s) 2024-09-18T14:19:11.547 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T14:19:11.572 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:19:11.641 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T14:19:11.686 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-18T14:19:11.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:19:11.789 INFO:teuthology.orchestra.run.smithi104.stdout:Reading package lists... 2024-09-18T14:19:11.859 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-18T14:19:11.859 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-18T14:19:12.008 INFO:teuthology.orchestra.run.smithi104.stdout:Building dependency tree... 2024-09-18T14:19:12.009 INFO:teuthology.orchestra.run.smithi104.stdout:Reading state information... 2024-09-18T14:19:12.058 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T14:19:12.059 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T14:19:12.059 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T14:19:12.059 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-09-18T14:19:12.059 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T14:19:12.059 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T14:19:12.060 INFO:teuthology.orchestra.run.smithi174.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T14:19:12.060 INFO:teuthology.orchestra.run.smithi174.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T14:19:12.060 INFO:teuthology.orchestra.run.smithi174.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T14:19:12.060 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T14:19:12.060 INFO:teuthology.orchestra.run.smithi174.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T14:19:12.060 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T14:19:12.061 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: socat xmlstarlet 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T14:19:12.062 INFO:teuthology.orchestra.run.smithi174.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T14:19:12.063 INFO:teuthology.orchestra.run.smithi174.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T14:19:12.063 INFO:teuthology.orchestra.run.smithi174.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T14:19:12.063 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-09-18T14:19:12.063 INFO:teuthology.orchestra.run.smithi174.stdout: btrfs-tools 2024-09-18T14:19:12.184 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-09-18T14:19:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T14:19:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T14:19:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T14:19:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T14:19:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T14:19:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T14:19:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T14:19:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T14:19:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T14:19:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T14:19:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T14:19:12.187 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2024-09-18T14:19:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 librbd1 2024-09-18T14:19:12.210 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T14:19:12.211 INFO:teuthology.orchestra.run.smithi104.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T14:19:12.211 INFO:teuthology.orchestra.run.smithi104.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T14:19:12.211 INFO:teuthology.orchestra.run.smithi104.stdout:The following additional packages will be installed: 2024-09-18T14:19:12.211 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T14:19:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T14:19:12.214 INFO:teuthology.orchestra.run.smithi104.stdout: socat xmlstarlet 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout:Suggested packages: 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout:Recommended packages: 2024-09-18T14:19:12.215 INFO:teuthology.orchestra.run.smithi104.stdout: btrfs-tools 2024-09-18T14:19:12.253 INFO:teuthology.orchestra.run.smithi174.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T14:19:12.253 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 201 MB of archives. 2024-09-18T14:19:12.253 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T14:19:12.254 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T14:19:12.332 INFO:teuthology.orchestra.run.smithi104.stdout:The following NEW packages will be installed: 2024-09-18T14:19:12.333 INFO:teuthology.orchestra.run.smithi104.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T14:19:12.333 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T14:19:12.333 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T14:19:12.333 INFO:teuthology.orchestra.run.smithi104.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T14:19:12.333 INFO:teuthology.orchestra.run.smithi104.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T14:19:12.333 INFO:teuthology.orchestra.run.smithi104.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T14:19:12.333 INFO:teuthology.orchestra.run.smithi104.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T14:19:12.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T14:19:12.336 INFO:teuthology.orchestra.run.smithi104.stdout:The following packages will be upgraded: 2024-09-18T14:19:12.337 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T14:19:12.337 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 librbd1 2024-09-18T14:19:12.337 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T14:19:12.338 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T14:19:12.346 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T14:19:12.349 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T14:19:12.376 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T14:19:12.378 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T14:19:12.383 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T14:19:12.415 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T14:19:12.417 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T14:19:12.426 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T14:19:12.429 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T14:19:12.429 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T14:19:12.431 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T14:19:12.432 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T14:19:12.432 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T14:19:12.433 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T14:19:12.434 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T14:19:12.449 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T14:19:12.450 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T14:19:12.450 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T14:19:12.450 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T14:19:12.466 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T14:19:12.466 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T14:19:12.467 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T14:19:12.469 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T14:19:12.470 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T14:19:12.473 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T14:19:12.483 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T14:19:12.483 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T14:19:12.487 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T14:19:12.487 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T14:19:12.497 INFO:teuthology.orchestra.run.smithi104.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T14:19:12.498 INFO:teuthology.orchestra.run.smithi104.stdout:Need to get 201 MB of archives. 2024-09-18T14:19:12.498 INFO:teuthology.orchestra.run.smithi104.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T14:19:12.498 INFO:teuthology.orchestra.run.smithi104.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T14:19:12.500 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T14:19:12.501 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T14:19:12.502 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T14:19:12.504 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T14:19:12.507 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T14:19:12.507 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T14:19:12.509 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T14:19:12.517 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T14:19:12.518 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T14:19:12.519 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T14:19:12.519 INFO:teuthology.orchestra.run.smithi174.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T14:19:12.529 INFO:teuthology.orchestra.run.smithi104.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T14:19:12.534 INFO:teuthology.orchestra.run.smithi174.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T14:19:12.560 INFO:teuthology.orchestra.run.smithi174.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T14:19:12.572 INFO:teuthology.orchestra.run.smithi174.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T14:19:12.579 INFO:teuthology.orchestra.run.smithi174.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T14:19:12.579 INFO:teuthology.orchestra.run.smithi174.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T14:19:12.582 INFO:teuthology.orchestra.run.smithi174.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T14:19:12.583 INFO:teuthology.orchestra.run.smithi174.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T14:19:12.591 INFO:teuthology.orchestra.run.smithi174.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T14:19:12.614 INFO:teuthology.orchestra.run.smithi174.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T14:19:12.629 INFO:teuthology.orchestra.run.smithi174.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T14:19:12.709 INFO:teuthology.orchestra.run.smithi174.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T14:19:12.710 INFO:teuthology.orchestra.run.smithi174.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T14:19:12.711 INFO:teuthology.orchestra.run.smithi174.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T14:19:12.718 INFO:teuthology.orchestra.run.smithi174.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T14:19:12.730 INFO:teuthology.orchestra.run.smithi174.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T14:19:12.731 INFO:teuthology.orchestra.run.smithi174.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T14:19:12.731 INFO:teuthology.orchestra.run.smithi174.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T14:19:12.731 INFO:teuthology.orchestra.run.smithi174.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T14:19:12.732 INFO:teuthology.orchestra.run.smithi174.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T14:19:12.732 INFO:teuthology.orchestra.run.smithi174.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T14:19:12.732 INFO:teuthology.orchestra.run.smithi174.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T14:19:12.758 INFO:teuthology.orchestra.run.smithi174.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T14:19:12.759 INFO:teuthology.orchestra.run.smithi174.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T14:19:12.759 INFO:teuthology.orchestra.run.smithi174.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T14:19:12.763 INFO:teuthology.orchestra.run.smithi174.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T14:19:12.765 INFO:teuthology.orchestra.run.smithi174.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T14:19:12.767 INFO:teuthology.orchestra.run.smithi174.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T14:19:12.768 INFO:teuthology.orchestra.run.smithi174.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T14:19:12.771 INFO:teuthology.orchestra.run.smithi174.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T14:19:12.774 INFO:teuthology.orchestra.run.smithi174.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T14:19:12.790 INFO:teuthology.orchestra.run.smithi174.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T14:19:12.792 INFO:teuthology.orchestra.run.smithi174.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T14:19:12.830 INFO:teuthology.orchestra.run.smithi174.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T14:19:12.831 INFO:teuthology.orchestra.run.smithi174.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T14:19:12.831 INFO:teuthology.orchestra.run.smithi174.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T14:19:12.831 INFO:teuthology.orchestra.run.smithi174.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T14:19:12.831 INFO:teuthology.orchestra.run.smithi174.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T14:19:12.832 INFO:teuthology.orchestra.run.smithi174.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T14:19:12.844 INFO:teuthology.orchestra.run.smithi174.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T14:19:12.844 INFO:teuthology.orchestra.run.smithi174.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T14:19:12.844 INFO:teuthology.orchestra.run.smithi174.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T14:19:12.847 INFO:teuthology.orchestra.run.smithi174.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T14:19:12.864 INFO:teuthology.orchestra.run.smithi174.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T14:19:12.865 INFO:teuthology.orchestra.run.smithi174.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T14:19:12.865 INFO:teuthology.orchestra.run.smithi174.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T14:19:12.867 INFO:teuthology.orchestra.run.smithi174.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T14:19:12.868 INFO:teuthology.orchestra.run.smithi174.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T14:19:12.931 INFO:teuthology.orchestra.run.smithi104.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T14:19:12.939 INFO:teuthology.orchestra.run.smithi104.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T14:19:12.982 INFO:teuthology.orchestra.run.smithi104.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T14:19:12.995 INFO:teuthology.orchestra.run.smithi104.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T14:19:13.122 INFO:teuthology.orchestra.run.smithi104.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T14:19:13.128 INFO:teuthology.orchestra.run.smithi104.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T14:19:13.157 INFO:teuthology.orchestra.run.smithi104.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T14:19:13.264 INFO:teuthology.orchestra.run.smithi174.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T14:19:13.298 INFO:teuthology.orchestra.run.smithi104.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T14:19:13.305 INFO:teuthology.orchestra.run.smithi104.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T14:19:13.310 INFO:teuthology.orchestra.run.smithi174.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T14:19:13.311 INFO:teuthology.orchestra.run.smithi174.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T14:19:13.313 INFO:teuthology.orchestra.run.smithi174.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T14:19:13.328 INFO:teuthology.orchestra.run.smithi104.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T14:19:13.335 INFO:teuthology.orchestra.run.smithi104.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T14:19:13.336 INFO:teuthology.orchestra.run.smithi104.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T14:19:13.337 INFO:teuthology.orchestra.run.smithi174.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T14:19:13.346 INFO:teuthology.orchestra.run.smithi104.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T14:19:13.347 INFO:teuthology.orchestra.run.smithi104.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T14:19:13.347 INFO:teuthology.orchestra.run.smithi104.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T14:19:13.348 INFO:teuthology.orchestra.run.smithi104.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T14:19:13.380 INFO:teuthology.orchestra.run.smithi104.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T14:19:13.422 INFO:teuthology.orchestra.run.smithi174.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T14:19:13.434 INFO:teuthology.orchestra.run.smithi104.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T14:19:13.435 INFO:teuthology.orchestra.run.smithi104.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T14:19:13.435 INFO:teuthology.orchestra.run.smithi104.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T14:19:13.436 INFO:teuthology.orchestra.run.smithi104.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T14:19:13.438 INFO:teuthology.orchestra.run.smithi104.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T14:19:13.438 INFO:teuthology.orchestra.run.smithi104.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T14:19:13.439 INFO:teuthology.orchestra.run.smithi104.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T14:19:13.441 INFO:teuthology.orchestra.run.smithi104.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T14:19:13.462 INFO:teuthology.orchestra.run.smithi104.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T14:19:13.506 INFO:teuthology.orchestra.run.smithi104.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T14:19:13.524 INFO:teuthology.orchestra.run.smithi104.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T14:19:13.526 INFO:teuthology.orchestra.run.smithi104.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T14:19:13.527 INFO:teuthology.orchestra.run.smithi104.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T14:19:13.535 INFO:teuthology.orchestra.run.smithi104.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T14:19:13.536 INFO:teuthology.orchestra.run.smithi104.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T14:19:13.537 INFO:teuthology.orchestra.run.smithi104.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T14:19:13.537 INFO:teuthology.orchestra.run.smithi104.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T14:19:13.544 INFO:teuthology.orchestra.run.smithi104.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T14:19:13.547 INFO:teuthology.orchestra.run.smithi104.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T14:19:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T14:19:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T14:19:13.629 INFO:teuthology.orchestra.run.smithi104.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T14:19:13.631 INFO:teuthology.orchestra.run.smithi104.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T14:19:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T14:19:13.633 INFO:teuthology.orchestra.run.smithi104.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T14:19:13.634 INFO:teuthology.orchestra.run.smithi104.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T14:19:13.914 INFO:teuthology.orchestra.run.smithi174.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T14:19:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T14:19:13.927 INFO:teuthology.orchestra.run.smithi174.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T14:19:13.944 INFO:teuthology.orchestra.run.smithi174.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T14:19:13.945 INFO:teuthology.orchestra.run.smithi174.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T14:19:13.946 INFO:teuthology.orchestra.run.smithi174.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T14:19:13.957 INFO:teuthology.orchestra.run.smithi104.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T14:19:14.032 INFO:teuthology.orchestra.run.smithi104.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T14:19:14.032 INFO:teuthology.orchestra.run.smithi104.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T14:19:14.034 INFO:teuthology.orchestra.run.smithi104.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T14:19:14.045 INFO:teuthology.orchestra.run.smithi174.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T14:19:14.056 INFO:teuthology.orchestra.run.smithi104.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T14:19:14.056 INFO:teuthology.orchestra.run.smithi104.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T14:19:14.056 INFO:teuthology.orchestra.run.smithi104.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T14:19:14.057 INFO:teuthology.orchestra.run.smithi104.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T14:19:14.058 INFO:teuthology.orchestra.run.smithi104.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T14:19:14.058 INFO:teuthology.orchestra.run.smithi104.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T14:19:14.058 INFO:teuthology.orchestra.run.smithi104.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T14:19:14.062 INFO:teuthology.orchestra.run.smithi104.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T14:19:14.119 INFO:teuthology.orchestra.run.smithi104.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T14:19:14.120 INFO:teuthology.orchestra.run.smithi104.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T14:19:14.121 INFO:teuthology.orchestra.run.smithi104.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T14:19:14.121 INFO:teuthology.orchestra.run.smithi104.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T14:19:14.147 INFO:teuthology.orchestra.run.smithi104.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T14:19:14.149 INFO:teuthology.orchestra.run.smithi104.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T14:19:14.196 INFO:teuthology.orchestra.run.smithi174.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T14:19:14.196 INFO:teuthology.orchestra.run.smithi174.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T14:19:14.203 INFO:teuthology.orchestra.run.smithi104.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T14:19:14.204 INFO:teuthology.orchestra.run.smithi104.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T14:19:14.206 INFO:teuthology.orchestra.run.smithi104.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T14:19:14.208 INFO:teuthology.orchestra.run.smithi104.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T14:19:14.211 INFO:teuthology.orchestra.run.smithi104.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T14:19:14.212 INFO:teuthology.orchestra.run.smithi104.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T14:19:14.216 INFO:teuthology.orchestra.run.smithi104.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T14:19:14.219 INFO:teuthology.orchestra.run.smithi104.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T14:19:14.292 INFO:teuthology.orchestra.run.smithi104.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T14:19:14.297 INFO:teuthology.orchestra.run.smithi104.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T14:19:14.461 INFO:teuthology.orchestra.run.smithi104.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T14:19:14.463 INFO:teuthology.orchestra.run.smithi104.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T14:19:14.463 INFO:teuthology.orchestra.run.smithi104.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T14:19:14.464 INFO:teuthology.orchestra.run.smithi104.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T14:19:14.464 INFO:teuthology.orchestra.run.smithi104.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T14:19:14.466 INFO:teuthology.orchestra.run.smithi104.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T14:19:14.477 INFO:teuthology.orchestra.run.smithi104.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T14:19:14.478 INFO:teuthology.orchestra.run.smithi104.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T14:19:14.478 INFO:teuthology.orchestra.run.smithi104.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T14:19:14.544 INFO:teuthology.orchestra.run.smithi104.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T14:19:14.625 INFO:teuthology.orchestra.run.smithi104.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T14:19:14.626 INFO:teuthology.orchestra.run.smithi104.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T14:19:14.626 INFO:teuthology.orchestra.run.smithi104.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T14:19:14.630 INFO:teuthology.orchestra.run.smithi104.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T14:19:14.632 INFO:teuthology.orchestra.run.smithi104.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T14:19:14.729 INFO:teuthology.orchestra.run.smithi104.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T14:19:14.732 INFO:teuthology.orchestra.run.smithi104.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T14:19:14.788 INFO:teuthology.orchestra.run.smithi104.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T14:19:15.526 INFO:teuthology.orchestra.run.smithi174.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T14:19:15.574 INFO:teuthology.orchestra.run.smithi174.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T14:19:15.575 INFO:teuthology.orchestra.run.smithi174.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T14:19:15.904 INFO:teuthology.orchestra.run.smithi104.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T14:19:15.974 INFO:teuthology.orchestra.run.smithi104.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T14:19:15.983 INFO:teuthology.orchestra.run.smithi104.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T14:19:15.985 INFO:teuthology.orchestra.run.smithi104.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T14:19:16.021 INFO:teuthology.orchestra.run.smithi104.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T14:19:16.068 INFO:teuthology.orchestra.run.smithi174.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T14:19:16.099 INFO:teuthology.orchestra.run.smithi104.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T14:19:16.554 INFO:teuthology.orchestra.run.smithi104.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T14:19:16.555 INFO:teuthology.orchestra.run.smithi104.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T14:19:16.566 INFO:teuthology.orchestra.run.smithi104.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T14:19:16.585 INFO:teuthology.orchestra.run.smithi104.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T14:19:16.585 INFO:teuthology.orchestra.run.smithi104.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T14:19:16.586 INFO:teuthology.orchestra.run.smithi104.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T14:19:16.682 INFO:teuthology.orchestra.run.smithi104.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T14:19:16.716 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 201 MB in 4s (51.9 MB/s) 2024-09-18T14:19:16.812 INFO:teuthology.orchestra.run.smithi104.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T14:19:16.812 INFO:teuthology.orchestra.run.smithi104.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T14:19:16.890 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T14:19:16.945 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T14:19:16.950 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T14:19:17.030 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:19:17.399 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:17.525 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:19:17.950 INFO:teuthology.orchestra.run.smithi104.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T14:19:17.955 INFO:teuthology.orchestra.run.smithi104.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T14:19:17.969 INFO:teuthology.orchestra.run.smithi104.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T14:19:18.211 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:18.281 INFO:teuthology.orchestra.run.smithi104.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T14:19:18.337 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:19:18.920 INFO:teuthology.orchestra.run.smithi104.stdout:Fetched 201 MB in 6s (33.9 MB/s) 2024-09-18T14:19:18.974 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T14:19:18.990 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:19.024 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:19.109 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T14:19:19.165 INFO:teuthology.orchestra.run.smithi104.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T14:19:19.169 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T14:19:19.282 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:19:19.335 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rados. 2024-09-18T14:19:19.351 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:19.384 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:19.652 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:19.737 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T14:19:19.755 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:19.761 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:19:19.795 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:20.081 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T14:19:20.098 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:20.139 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:20.432 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:20.458 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T14:19:20.475 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:20.526 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:20.573 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:19:20.803 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T14:19:20.821 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T14:19:20.852 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T14:19:21.064 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T14:19:21.082 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:21.122 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:21.252 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T14:19:21.268 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:21.310 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:21.499 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T14:19:21.515 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T14:19:21.557 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:19:21.672 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rados. 2024-09-18T14:19:21.688 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:21.730 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:21.927 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T14:19:21.943 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T14:19:21.977 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:19:22.099 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T14:19:22.118 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:22.191 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:22.246 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T14:19:22.265 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T14:19:22.296 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:19:22.409 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T14:19:22.427 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:22.493 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:22.691 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T14:19:22.707 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T14:19:22.749 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:19:22.846 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T14:19:22.861 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:22.904 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:23.085 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T14:19:23.100 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T14:19:23.143 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:19:23.257 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T14:19:23.272 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T14:19:23.315 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T14:19:23.601 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T14:19:23.619 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:23.621 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T14:19:23.641 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:19:23.659 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:23.690 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:24.067 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T14:19:24.086 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T14:19:24.137 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:19:24.208 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T14:19:24.223 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:19:24.266 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:24.465 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T14:19:24.482 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T14:19:24.523 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:19:24.635 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T14:19:24.653 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:19:24.693 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:24.876 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T14:19:24.894 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T14:19:24.934 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:19:25.088 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T14:19:25.106 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T14:19:25.146 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:19:25.370 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T14:19:25.386 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T14:19:25.428 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:19:25.482 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librgw2. 2024-09-18T14:19:25.499 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:25.532 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:25.740 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T14:19:25.756 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T14:19:25.789 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:19:26.301 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T14:19:26.307 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T14:19:26.318 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:26.326 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:19:26.359 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:26.386 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:26.721 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liboath0. 2024-09-18T14:19:26.736 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T14:19:26.780 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T14:19:26.913 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T14:19:26.928 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:19:26.979 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:27.090 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T14:19:27.107 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:27.148 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:27.324 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T14:19:27.342 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:19:27.374 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:27.618 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-common. 2024-09-18T14:19:27.637 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:27.676 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:27.810 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T14:19:27.827 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T14:19:27.868 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:19:28.246 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package librgw2. 2024-09-18T14:19:28.262 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:28.304 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:29.073 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T14:19:29.092 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:29.123 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:29.526 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liboath0. 2024-09-18T14:19:29.542 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T14:19:29.584 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T14:19:29.904 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T14:19:29.923 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:29.962 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:30.206 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-base. 2024-09-18T14:19:30.225 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:30.274 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:30.607 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-common. 2024-09-18T14:19:30.623 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:30.657 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:31.168 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T14:19:31.186 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T14:19:31.226 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:19:31.647 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T14:19:31.666 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T14:19:31.705 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T14:19:32.016 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T14:19:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T14:19:32.099 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T14:19:32.351 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T14:19:32.367 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:32.409 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:32.822 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T14:19:32.842 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T14:19:32.880 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:19:33.087 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-base. 2024-09-18T14:19:33.106 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:33.156 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:33.183 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T14:19:33.199 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T14:19:33.241 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:19:33.578 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T14:19:33.594 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T14:19:33.627 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T14:19:33.872 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T14:19:33.890 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T14:19:33.922 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:19:34.091 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T14:19:34.108 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T14:19:34.149 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:19:34.242 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T14:19:34.261 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T14:19:34.300 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T14:19:34.572 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T14:19:34.586 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T14:19:34.588 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T14:19:34.605 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T14:19:34.643 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:19:34.663 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T14:19:34.941 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T14:19:34.959 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T14:19:34.980 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-mako. 2024-09-18T14:19:34.996 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T14:19:35.007 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T14:19:35.038 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:19:35.260 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T14:19:35.276 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:35.333 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T14:19:35.334 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:35.350 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T14:19:35.382 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:19:35.693 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T14:19:35.711 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T14:19:35.731 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T14:19:35.748 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T14:19:35.751 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:19:35.789 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:19:36.038 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webob. 2024-09-18T14:19:36.055 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T14:19:36.096 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:19:36.117 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T14:19:36.133 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T14:19:36.175 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:19:36.448 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T14:19:36.469 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T14:19:36.506 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:19:36.536 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T14:19:36.553 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T14:19:36.612 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T14:19:36.834 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T14:19:36.850 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T14:19:36.892 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T14:19:36.956 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T14:19:36.972 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T14:19:37.014 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:19:37.246 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T14:19:37.262 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T14:19:37.307 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:19:37.351 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T14:19:37.368 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T14:19:37.409 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T14:19:37.657 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T14:19:37.673 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T14:19:37.715 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T14:19:37.737 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T14:19:37.752 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T14:19:37.795 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:19:37.993 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-paste. 2024-09-18T14:19:38.011 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T14:19:38.051 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:19:38.123 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-mako. 2024-09-18T14:19:38.142 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T14:19:38.214 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:19:38.417 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T14:19:38.435 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T14:19:38.476 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:19:38.550 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T14:19:38.568 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T14:19:38.609 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:19:38.762 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T14:19:38.781 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T14:19:38.820 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:19:38.919 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T14:19:38.938 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T14:19:38.977 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:19:39.148 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T14:19:39.164 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T14:19:39.206 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T14:19:39.314 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webob. 2024-09-18T14:19:39.330 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T14:19:39.372 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:19:39.492 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T14:19:39.510 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T14:19:39.572 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:19:39.733 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T14:19:39.751 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T14:19:39.816 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:19:39.896 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T14:19:39.912 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T14:19:39.954 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:19:40.141 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T14:19:40.157 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T14:19:40.199 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T14:19:40.308 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T14:19:40.326 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:40.366 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:40.560 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T14:19:40.579 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T14:19:40.646 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:19:40.711 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T14:19:40.727 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:40.769 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:41.013 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T14:19:41.029 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T14:19:41.096 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T14:19:41.240 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T14:19:41.258 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:41.298 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:41.424 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-paste. 2024-09-18T14:19:41.443 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T14:19:41.482 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:19:41.827 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T14:19:41.846 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T14:19:41.885 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:19:42.154 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T14:19:42.170 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:42.171 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T14:19:42.190 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T14:19:42.211 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:42.221 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:19:42.560 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T14:19:42.578 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T14:19:42.616 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T14:19:42.960 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T14:19:42.976 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T14:19:43.051 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:19:43.402 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T14:19:43.420 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T14:19:43.451 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:19:43.838 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T14:19:43.854 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:43.888 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:44.306 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph. 2024-09-18T14:19:44.314 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T14:19:44.323 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:44.330 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:44.371 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:44.372 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:44.699 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T14:19:44.715 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:44.757 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:44.875 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T14:19:44.892 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:44.975 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:45.927 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T14:19:45.943 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:45.984 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:46.496 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephadm. 2024-09-18T14:19:46.514 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:46.545 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:46.596 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T14:19:46.615 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:46.678 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:46.866 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T14:19:46.883 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T14:19:46.924 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:19:47.268 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T14:19:47.284 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:47.326 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:47.671 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T14:19:47.687 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T14:19:47.729 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T14:19:47.998 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T14:19:48.015 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T14:19:48.056 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:19:48.317 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T14:19:48.334 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T14:19:48.367 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:19:48.686 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-routes. 2024-09-18T14:19:48.705 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T14:19:48.769 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T14:19:48.807 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph. 2024-09-18T14:19:48.827 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:48.857 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:49.097 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T14:19:49.113 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:49.168 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T14:19:49.171 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:49.184 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:49.244 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:49.687 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T14:19:49.704 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:49.745 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:50.282 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package cephadm. 2024-09-18T14:19:50.302 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:50.340 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:50.402 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T14:19:50.421 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T14:19:50.478 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:19:50.643 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T14:19:50.663 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T14:19:50.701 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:19:50.813 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T14:19:50.828 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T14:19:50.870 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:19:51.054 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T14:19:51.071 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:51.112 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:51.215 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T14:19:51.231 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T14:19:51.273 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T14:19:51.515 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T14:19:51.534 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T14:19:51.577 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T14:19:51.901 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T14:19:51.917 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T14:19:51.959 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:19:52.253 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T14:19:52.270 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T14:19:52.311 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:19:52.599 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T14:19:52.618 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T14:19:52.647 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-routes. 2024-09-18T14:19:52.657 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:19:52.667 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T14:19:52.705 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T14:19:53.049 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T14:19:53.065 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:53.124 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:53.202 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T14:19:53.222 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T14:19:53.260 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T14:19:53.630 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T14:19:53.647 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T14:19:53.688 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:19:54.429 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T14:19:54.447 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:54.481 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T14:19:54.487 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:54.498 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T14:19:54.538 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:19:54.866 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T14:19:54.882 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T14:19:54.924 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:19:55.227 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T14:19:55.244 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T14:19:55.285 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T14:19:55.549 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T14:19:55.567 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T14:19:55.615 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T14:19:55.918 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T14:19:55.935 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T14:19:55.984 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:19:56.321 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T14:19:56.340 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T14:19:56.378 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:19:56.685 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T14:19:56.700 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T14:19:56.704 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T14:19:56.718 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T14:19:56.743 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:19:56.756 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T14:19:57.059 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T14:19:57.076 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T14:19:57.117 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T14:19:57.339 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T14:19:57.356 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T14:19:57.397 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T14:19:57.412 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T14:19:57.428 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T14:19:57.461 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:19:57.764 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T14:19:57.780 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T14:19:57.792 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T14:19:57.808 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T14:19:57.822 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:19:57.849 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:19:58.168 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T14:19:58.184 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T14:19:58.225 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:19:58.554 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T14:19:58.571 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T14:19:58.612 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:19:58.641 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T14:19:58.657 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:58.699 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:58.940 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T14:19:58.958 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T14:19:58.998 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:19:59.639 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T14:19:59.658 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:59.697 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:59.810 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T14:19:59.826 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T14:19:59.885 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T14:20:00.033 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T14:20:00.049 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T14:20:00.091 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:20:00.221 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T14:20:00.237 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T14:20:00.287 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:20:00.453 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T14:20:00.471 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:20:00.510 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:00.657 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T14:20:00.673 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T14:20:00.715 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:20:00.824 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package jq. 2024-09-18T14:20:00.840 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:20:00.883 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:01.043 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T14:20:01.063 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T14:20:01.126 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T14:20:01.389 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T14:20:01.405 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T14:20:01.463 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T14:20:01.528 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package socat. 2024-09-18T14:20:01.544 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T14:20:01.586 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T14:20:01.800 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T14:20:01.817 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T14:20:01.889 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T14:20:01.907 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T14:20:01.939 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:20:01.966 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:20:02.319 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-test. 2024-09-18T14:20:02.336 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:02.376 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:02.485 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T14:20:02.501 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T14:20:02.568 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:20:02.923 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T14:20:02.939 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T14:20:02.981 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:20:03.317 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T14:20:03.333 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T14:20:03.375 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:20:03.678 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T14:20:03.697 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T14:20:03.727 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:20:04.405 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T14:20:04.424 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:20:04.488 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:04.832 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T14:20:04.849 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T14:20:04.890 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:20:05.268 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T14:20:05.285 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:20:05.326 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:05.612 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package jq. 2024-09-18T14:20:05.631 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:20:05.670 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:05.949 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package socat. 2024-09-18T14:20:05.965 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T14:20:06.007 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T14:20:06.364 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T14:20:06.383 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T14:20:06.427 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:20:06.812 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-test. 2024-09-18T14:20:06.829 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:06.839 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T14:20:06.856 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:20:06.870 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:06.897 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:07.213 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T14:20:07.229 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:20:07.271 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:07.742 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T14:20:07.759 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:07.800 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:08.221 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T14:20:08.237 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T14:20:08.297 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:20:08.617 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T14:20:08.637 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:20:08.675 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:10.366 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T14:20:10.387 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T14:20:10.424 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T14:20:10.760 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T14:20:10.777 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T14:20:10.818 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:20:11.138 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-attr. 2024-09-18T14:20:11.158 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T14:20:11.196 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T14:20:11.432 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T14:20:11.451 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:20:11.481 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:11.490 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T14:20:11.510 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T14:20:11.573 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:20:11.863 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T14:20:11.876 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T14:20:11.883 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:20:11.892 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T14:20:11.921 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:11.933 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T14:20:12.278 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T14:20:12.299 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T14:20:12.336 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:20:12.425 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T14:20:12.444 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:12.508 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:12.773 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T14:20:12.793 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T14:20:12.831 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T14:20:12.903 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T14:20:12.919 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T14:20:12.978 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:20:13.176 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T14:20:13.193 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T14:20:13.234 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T14:20:13.307 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T14:20:13.327 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:20:13.365 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:13.545 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T14:20:13.564 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T14:20:13.628 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:20:13.956 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-py. 2024-09-18T14:20:13.975 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T14:20:14.014 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:20:14.359 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T14:20:14.376 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T14:20:14.417 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:20:14.719 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T14:20:14.737 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T14:20:14.777 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:20:14.972 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T14:20:14.991 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T14:20:15.055 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T14:20:15.097 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T14:20:15.117 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T14:20:15.146 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T14:20:15.400 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T14:20:15.420 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T14:20:15.458 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:20:15.483 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T14:20:15.501 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T14:20:15.541 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:20:15.744 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-attr. 2024-09-18T14:20:15.761 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T14:20:15.794 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T14:20:15.827 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T14:20:15.847 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T14:20:15.885 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:20:16.138 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T14:20:16.157 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T14:20:16.196 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:20:16.507 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T14:20:16.528 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T14:20:16.565 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T14:20:16.606 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package radosgw. 2024-09-18T14:20:16.626 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:16.664 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:16.927 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T14:20:16.945 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T14:20:17.002 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:20:17.513 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T14:20:17.532 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T14:20:17.571 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T14:20:17.950 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T14:20:17.966 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T14:20:18.016 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T14:20:18.344 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T14:20:18.361 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T14:20:18.367 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T14:20:18.384 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:18.402 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:20:18.425 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:18.740 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T14:20:18.763 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-py. 2024-09-18T14:20:18.781 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T14:20:18.821 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:20:19.033 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:20:19.159 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:20:19.174 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T14:20:19.193 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T14:20:19.249 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:20:19.285 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:20:19.486 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:19.568 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T14:20:19.588 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T14:20:19.626 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:20:19.954 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T14:20:19.970 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T14:20:20.012 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T14:20:20.073 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user cephadm....done 2024-09-18T14:20:20.234 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:20.366 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T14:20:20.382 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T14:20:20.384 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:20:20.418 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:20:20.721 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T14:20:20.736 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T14:20:20.738 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T14:20:20.801 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:20:21.046 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:21.174 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T14:20:21.300 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:20:21.566 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package radosgw. 2024-09-18T14:20:21.585 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:21.624 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:21.650 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T14:20:22.112 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:20:22.413 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T14:20:22.689 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:20:22.965 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:20:23.283 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:23.352 INFO:teuthology.orchestra.run.smithi104.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T14:20:23.370 INFO:teuthology.orchestra.run.smithi104.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:20:23.435 INFO:teuthology.orchestra.run.smithi104.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:23.575 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:20:23.727 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T14:20:23.735 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:20:24.028 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:20:24.032 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:20:24.177 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:20:24.302 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:20:24.355 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:20:24.480 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:20:24.521 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:24.848 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T14:20:25.124 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:20:25.149 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user cephadm....done 2024-09-18T14:20:25.310 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:25.408 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:20:25.453 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:20:25.768 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:20:25.805 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T14:20:26.052 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:20:26.098 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:20:26.232 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T14:20:26.392 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:20:26.395 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T14:20:26.688 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T14:20:26.718 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T14:20:26.997 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:20:27.128 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:20:27.131 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:20:27.249 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:20:27.421 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T14:20:27.542 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:20:27.714 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:20:27.943 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:20:27.981 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:20:28.283 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:28.303 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:20:28.584 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:20:28.688 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T14:20:28.712 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:20:28.814 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:28.994 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:20:29.123 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:20:29.257 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T14:20:29.278 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:20:29.421 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:20:29.592 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:20:29.718 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:20:29.826 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T14:20:29.869 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T14:20:30.145 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:20:30.170 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:20:30.424 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:20:30.480 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:20:30.606 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:20:30.732 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:20:30.817 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:20:31.041 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:20:31.126 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:20:31.342 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T14:20:31.469 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T14:20:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:20:31.780 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T14:20:31.802 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:20:32.087 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:20:32.213 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:20:32.339 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:20:32.429 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T14:20:32.657 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:20:32.813 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:20:33.106 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:20:33.300 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:20:33.466 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T14:20:33.702 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:20:33.725 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:20:33.892 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:20:34.062 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T14:20:34.179 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:34.201 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:20:34.514 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:20:34.545 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:20:34.671 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:20:34.673 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T14:20:34.797 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:20:35.024 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:20:35.123 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:20:35.150 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:20:35.293 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T14:20:35.415 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:35.541 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T14:20:35.611 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:20:35.945 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:20:36.072 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:20:36.197 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:20:36.574 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:20:36.883 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T14:20:36.996 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:37.122 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:37.217 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:20:37.265 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T14:20:37.335 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:20:37.619 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T14:20:37.625 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T14:20:37.920 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:20:37.926 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:20:38.210 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:20:38.221 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:20:38.345 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T14:20:38.585 INFO:teuthology.orchestra.run.smithi104.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T14:20:38.696 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:38.822 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:20:38.874 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:20:39.000 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:20:39.149 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:39.342 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:20:39.357 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:20:39.692 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:20:39.752 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:20:39.878 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:20:40.004 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:20:40.405 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:20:40.714 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:40.731 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:40.840 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:20:40.848 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T14:20:40.950 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T14:20:41.075 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:41.201 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T14:20:41.527 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:20:41.653 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T14:20:41.979 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:42.105 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:42.321 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:42.330 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:20:42.330 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:20:42.446 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:42.572 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T14:20:42.806 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:42.984 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T14:20:43.026 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:20:43.302 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:20:43.603 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T14:20:43.628 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:20:43.754 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T14:20:43.997 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:20:44.113 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:44.256 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:20:44.349 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:44.466 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:44.591 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:44.592 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:44.717 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:20:44.743 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:44.869 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T14:20:45.052 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:20:45.171 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T14:20:45.489 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:45.615 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:45.716 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:20:46.082 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:46.241 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:20:46.367 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T14:20:46.518 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:20:46.644 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T14:20:46.652 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:46.937 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:20:47.062 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T14:20:47.196 INFO:teuthology.orchestra.run.smithi174.stdout:Adding group ceph....done 2024-09-18T14:20:47.397 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:47.548 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:47.597 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user ceph....done 2024-09-18T14:20:47.773 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:20:47.773 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:20:47.774 INFO:teuthology.orchestra.run.smithi174.stdout:Setting system user ceph properties....done 2024-09-18T14:20:47.797 INFO:teuthology.orchestra.run.smithi174.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T14:20:47.945 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T14:20:48.216 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:48.254 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T14:20:48.477 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:20:48.728 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:48.771 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T14:20:48.846 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:49.080 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:20:49.133 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:20:49.133 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:20:49.425 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:49.551 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:49.702 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:49.722 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:49.828 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:49.945 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T14:20:50.105 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T14:20:50.300 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T14:20:50.541 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:50.632 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:50.764 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:20:50.765 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:20:50.766 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:50.892 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:20:51.209 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:51.436 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:20:51.436 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:20:51.837 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:51.877 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:52.362 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:20:52.362 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:20:52.441 INFO:teuthology.orchestra.run.smithi104.stdout:Adding group ceph....done 2024-09-18T14:20:52.803 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:52.866 INFO:teuthology.orchestra.run.smithi104.stdout:Adding system user ceph....done 2024-09-18T14:20:52.904 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:53.058 INFO:teuthology.orchestra.run.smithi104.stdout:Setting system user ceph properties....done 2024-09-18T14:20:53.079 INFO:teuthology.orchestra.run.smithi104.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T14:20:53.230 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T14:20:53.257 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:53.480 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:20:53.480 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:20:53.557 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T14:20:53.924 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:54.013 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:54.100 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:54.139 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:54.226 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:54.369 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:54.432 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:20:54.432 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:20:54.498 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T14:20:54.649 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T14:20:54.959 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T14:20:55.091 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:55.490 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T14:20:55.951 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:56.174 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:20:56.175 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:20:56.619 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:56.843 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:20:56.844 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:20:57.286 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:57.778 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T14:20:57.805 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:20:57.805 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:20:58.288 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:58.393 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T14:20:58.398 DEBUG:teuthology.parallel:result is None 2024-09-18T14:20:58.447 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:58.590 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:58.988 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:20:58.989 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:20:59.434 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:59.577 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:59.745 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:20:59.887 INFO:teuthology.orchestra.run.smithi104.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:21:00.039 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T14:21:00.190 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T14:21:00.517 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T14:21:03.352 INFO:teuthology.orchestra.run.smithi104.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T14:21:03.957 INFO:teuthology.orchestra.run.smithi104.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T14:21:03.963 DEBUG:teuthology.parallel:result is None 2024-09-18T14:21:03.964 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:21:04.104 DEBUG:teuthology.orchestra.run.smithi104:> dpkg-query -W -f '${Version}' ceph 2024-09-18T14:21:04.127 INFO:teuthology.orchestra.run.smithi104.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:21:04.128 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:21:04.128 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T14:21:04.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:21:04.269 DEBUG:teuthology.orchestra.run.smithi174:> dpkg-query -W -f '${Version}' ceph 2024-09-18T14:21:04.293 INFO:teuthology.orchestra.run.smithi174.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:21:04.294 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:21:04.294 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T14:21:04.296 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T14:21:04.296 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:04.296 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T14:21:04.313 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:04.313 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T14:21:04.328 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T14:21:04.329 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:04.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T14:21:04.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T14:21:04.399 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:04.399 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T14:21:04.414 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T14:21:04.424 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T14:21:04.424 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:04.424 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T14:21:04.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T14:21:04.497 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:04.497 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T14:21:04.513 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T14:21:04.572 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T14:21:04.572 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:04.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T14:21:04.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T14:21:04.600 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:04.600 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T14:21:04.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T14:21:04.625 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T14:21:04.707 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T14:21:04.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2024-09-18T14:21:04.804 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2024-09-18T14:21:04.826 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T14:21:04.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T14:21:04.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T14:21:04.886 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T14:21:04.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T14:21:04.924 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T14:21:05.025 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T14:21:05.025 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T14:21:05.025 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101', 'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/pool-create-delete}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7911101', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'workunit': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 4631, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'targets': {'smithi104.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi174.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_09:13:00', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T14:21:05.026 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T14:21:05.028 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T14:21:05.036 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T14:21:05.038 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T14:21:05.051 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:05.051 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T14:21:05.096 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T14:21:05.096 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 359 Links: 1 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-18 14:17:36.985019514 +0000 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-18 14:17:36.781024209 +0000 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-18 14:17:36.781024209 +0000 2024-09-18T14:21:05.104 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-09-18T14:21:05.105 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T14:21:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-18T14:21:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-18T14:21:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000423898 s, 1.2 MB/s 2024-09-18T14:21:05.118 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T14:21:05.126 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 366 Links: 1 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-18 14:17:37.389010220 +0000 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-18 14:17:37.205014453 +0000 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-18 14:17:37.205014453 +0000 2024-09-18T14:21:05.181 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-09-18T14:21:05.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T14:21:05.194 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-18T14:21:05.194 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-18T14:21:05.194 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000414011 s, 1.2 MB/s 2024-09-18T14:21:05.195 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T14:21:05.202 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-09-18T14:21:05.208 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T14:21:05.208 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.208 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-09-18T14:21:05.208 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.208 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-18 14:17:37.880998900 +0000 2024-09-18T14:21:05.208 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-18 14:17:37.681003502 +0000 2024-09-18T14:21:05.209 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-18 14:17:37.681003502 +0000 2024-09-18T14:21:05.209 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-09-18T14:21:05.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T14:21:05.219 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-18T14:21:05.220 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-18T14:21:05.220 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000389566 s, 1.3 MB/s 2024-09-18T14:21:05.221 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T14:21:05.274 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 380 Links: 1 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-18 14:17:38.328988593 +0000 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-18 14:17:38.108993655 +0000 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-18 14:17:38.108993655 +0000 2024-09-18T14:21:05.281 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2024-09-18T14:21:05.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T14:21:05.293 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-18T14:21:05.293 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-18T14:21:05.293 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000361519 s, 1.4 MB/s 2024-09-18T14:21:05.294 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T14:21:05.346 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:05.346 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:05.346 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T14:21:05.352 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T14:21:05.353 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-09-18T14:21:05.359 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T14:21:05.360 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.360 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 368 Links: 1 2024-09-18T14:21:05.360 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.360 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-18 14:17:34.771813057 +0000 2024-09-18T14:21:05.360 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-18 14:17:34.767813149 +0000 2024-09-18T14:21:05.360 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-18 14:17:34.767813149 +0000 2024-09-18T14:21:05.360 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-09-18T14:21:05.360 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T14:21:05.370 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-18T14:21:05.370 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-18T14:21:05.370 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000431905 s, 1.2 MB/s 2024-09-18T14:21:05.371 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T14:21:05.420 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 375 Links: 1 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-18 14:17:35.403798510 +0000 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-18 14:17:35.219802745 +0000 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-18 14:17:35.219802745 +0000 2024-09-18T14:21:05.427 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-09-18T14:21:05.427 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T14:21:05.437 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-18T14:21:05.438 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-18T14:21:05.438 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000432084 s, 1.2 MB/s 2024-09-18T14:21:05.439 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T14:21:05.489 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-09-18T14:21:05.496 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T14:21:05.496 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.496 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 383 Links: 1 2024-09-18T14:21:05.496 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.497 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-18 14:17:35.827788749 +0000 2024-09-18T14:21:05.497 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-18 14:17:35.631793262 +0000 2024-09-18T14:21:05.497 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-18 14:17:35.631793262 +0000 2024-09-18T14:21:05.497 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-09-18T14:21:05.497 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T14:21:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-18T14:21:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-18T14:21:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000397984 s, 1.3 MB/s 2024-09-18T14:21:05.508 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T14:21:05.515 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-09-18T14:21:05.563 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T14:21:05.563 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:21:05.563 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 387 Links: 1 2024-09-18T14:21:05.563 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:21:05.564 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-18 14:17:36.231779450 +0000 2024-09-18T14:21:05.564 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-18 14:17:36.035783963 +0000 2024-09-18T14:21:05.564 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-18 14:17:36.035783963 +0000 2024-09-18T14:21:05.564 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-09-18T14:21:05.564 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T14:21:05.575 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-18T14:21:05.575 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-18T14:21:05.575 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000365462 s, 1.4 MB/s 2024-09-18T14:21:05.576 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T14:21:05.583 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:05.583 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T14:21:05.583 INFO:tasks.ceph:Generating config... 2024-09-18T14:21:05.585 INFO:tasks.ceph:[client] debug monc = 20 2024-09-18T14:21:05.585 INFO:tasks.ceph:[client] debug ms = 1 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-18T14:21:05.585 INFO:tasks.ceph:[global] ms type = async 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-18T14:21:05.586 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-18T14:21:05.587 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-18T14:21:05.587 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-18T14:21:05.587 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd objectstore = filestore 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-18T14:21:05.587 INFO:tasks.ceph:[osd] osd sloppy crc = True 2024-09-18T14:21:05.588 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T14:21:05.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T14:21:05.623 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T14:21:05.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T14:21:05.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T14:21:05.722 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.104'), ('mon.c', '[v2:172.21.15.104:3301,v1:172.21.15.104:6790]'), ('mon.b', '172.21.15.174')] 2024-09-18T14:21:05.722 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.104,[v2:172.21.15.104:3301,v1:172.21.15.104:6790],172.21.15.174', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-18T14:21:05.723 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:05.723 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T14:21:05.728 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.104 --addv c '[v2:172.21.15.104:3301,v1:172.21.15.104:6790]' --add b 172.21.15.174 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:05.808 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: generated fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:setting min_mon_release = octopus 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:epoch 0 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:min_mon_release 15 (octopus) 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:election_strategy: 1 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:05.809 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T14:21:05.810 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T14:21:05.815 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8... 2024-09-18T14:21:05.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T14:21:05.818 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T14:21:05.849 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-09-18T14:21:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2024-09-18T14:21:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T14:21:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2024-09-18T14:21:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2024-09-18T14:21:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.850 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.851 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T14:21:05.852 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2024-09-18T14:21:05.852 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.852 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T14:21:05.852 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T14:21:05.852 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.852 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2024-09-18T14:21:05.853 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.853 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2024-09-18T14:21:05.853 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2024-09-18T14:21:05.853 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2024-09-18T14:21:05.853 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2024-09-18T14:21:05.853 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2024-09-18T14:21:05.854 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2024-09-18T14:21:05.854 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2024-09-18T14:21:05.854 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T14:21:05.854 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T14:21:05.854 INFO:teuthology.orchestra.run.smithi104.stdout: mon_allow_pool_size_one = true 2024-09-18T14:21:05.854 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.854 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T14:21:05.855 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.855 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2024-09-18T14:21:05.855 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.855 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2024-09-18T14:21:05.855 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.856 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2024-09-18T14:21:05.856 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2024-09-18T14:21:05.856 INFO:teuthology.orchestra.run.smithi104.stdout: mon health detail to clog = false 2024-09-18T14:21:05.856 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "172.21.15.104,[v2:172.21.15.104:3301,v1:172.21.15.104:6790],172.21.15.174" 2024-09-18T14:21:05.856 INFO:teuthology.orchestra.run.smithi104.stdout: mon client directed command retry = 5 2024-09-18T14:21:05.856 INFO:teuthology.orchestra.run.smithi104.stdout: mon election default strategy = 1 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms bind msgr1 = False 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms bind msgr2 = True 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject delay max = 1 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject delay probability = 0.005 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject delay type = mon 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject internal delays = 0.002 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject socket failures = 2500 2024-09-18T14:21:05.857 INFO:teuthology.orchestra.run.smithi104.stdout: ms type = async 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 600 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock profile = high_recovery_ops 2024-09-18T14:21:05.858 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2024-09-18T14:21:05.859 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore zero block detection = True 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 20 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify cached snaps = True 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify missing on start = True 2024-09-18T14:21:05.860 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock override recovery settings = True 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = filestore 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue = debug_random 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue cut off = debug_random 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: osd sloppy crc = True 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-09-18T14:21:05.861 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 10 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2024-09-18T14:21:05.862 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: # 1m isn't quite enough 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: mon_down_mkfs_grace = 2m 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_filestore_osds = false 2024-09-18T14:21:05.863 INFO:teuthology.orchestra.run.smithi104.stdout: mon min osdmap epochs = 25 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune interval = 2 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune min = 15 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune txsize = 2 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub inject missing keys = 0.05 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub interval = 300 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: paxos min = 10 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: paxos service trim min = 5 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: paxos trim min = 10 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2024-09-18T14:21:05.864 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2024-09-18T14:21:05.865 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2024-09-18T14:21:05.867 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2024-09-18T14:21:05.867 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2024-09-18T14:21:05.867 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T14:21:05.867 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.868 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2024-09-18T14:21:05.869 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: mon_allow_pool_size_one = true 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2024-09-18T14:21:05.870 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.104,[v2:172.21.15.104:3301,v1:172.21.15.104:6790],172.21.15.174" 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: mon client directed command retry = 5 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: mon election default strategy = 1 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: ms bind msgr1 = False 2024-09-18T14:21:05.871 INFO:teuthology.orchestra.run.smithi174.stdout: ms bind msgr2 = True 2024-09-18T14:21:05.872 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay max = 1 2024-09-18T14:21:05.872 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay probability = 0.005 2024-09-18T14:21:05.872 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay type = mon 2024-09-18T14:21:05.872 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject internal delays = 0.002 2024-09-18T14:21:05.872 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject socket failures = 2500 2024-09-18T14:21:05.872 INFO:teuthology.orchestra.run.smithi174.stdout: ms type = async 2024-09-18T14:21:05.872 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock profile = high_recovery_ops 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2024-09-18T14:21:05.873 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2024-09-18T14:21:05.874 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore zero block detection = True 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify cached snaps = True 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify missing on start = True 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock override recovery settings = True 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = filestore 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd op queue = debug_random 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd op queue cut off = debug_random 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: osd sloppy crc = True 2024-09-18T14:21:05.875 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-09-18T14:21:05.876 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout: debug monc = 10 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-09-18T14:21:05.877 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.878 INFO:teuthology.orchestra.run.smithi174.stdout: # 1m isn't quite enough 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: mon_down_mkfs_grace = 2m 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_filestore_osds = false 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: mon min osdmap epochs = 25 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: mon osdmap full prune interval = 2 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: mon osdmap full prune min = 15 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: mon osdmap full prune txsize = 2 2024-09-18T14:21:05.879 INFO:teuthology.orchestra.run.smithi174.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: mon scrub inject missing keys = 0.05 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: mon scrub interval = 300 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: paxos min = 10 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: paxos service trim min = 5 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: paxos trim min = 10 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2024-09-18T14:21:05.880 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout: debug monc = 20 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2024-09-18T14:21:05.881 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2024-09-18T14:21:05.882 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T14:21:05.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T14:21:05.958 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T14:21:05.958 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:05.958 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T14:21:05.965 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:05.965 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T14:21:05.970 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:21:05.970 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:05.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T14:21:05.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T14:21:05.990 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:05.991 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:06.040 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:21:06.040 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:06.040 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T14:21:06.040 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T14:21:06.068 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:06.069 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:06.075 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T14:21:06.075 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T14:21:06.076 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T14:21:06.161 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T14:21:06.164 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T14:21:06.164 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T14:21:06.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-18T14:21:06.197 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-18T14:21:06.210 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T14:21:06.211 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T14:21:06.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T14:21:06.227 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:06.227 INFO:tasks.ceph:role: osd.0 2024-09-18T14:21:06.227 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:21:06.227 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:21:06.803 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:21:06.804 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:21:06.804 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:21:06.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-09-18T14:21:06.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T14:21:06.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T14:21:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:21:06.915 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:21:06.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T14:21:06.927 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:06.927 INFO:tasks.ceph:role: osd.1 2024-09-18T14:21:06.927 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:21:06.927 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:21:07.495 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:21:07.497 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-09-18T14:21:07.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T14:21:07.531 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T14:21:07.539 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:21:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:21:07.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T14:21:07.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:07.597 INFO:tasks.ceph:role: osd.2 2024-09-18T14:21:07.597 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:21:07.597 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T14:21:08.085 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:21:08.086 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:21:08.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-09-18T14:21:08.088 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T14:21:08.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T14:21:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:21:08.144 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:21:08.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:08.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.253+0000 7efd5b94a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T14:21:08.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.253+0000 7efd5b94a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T14:21:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.277+0000 7efd5b94a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.277+0000 7efd5b94a540 -1 journal FileJournal::_open_file : zeroing journal 2024-09-18T14:21:08.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.557+0000 7efd5b94a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:08.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.561+0000 7efd5b94a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:08.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.561+0000 7efd5b94a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:08.594 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T14:21:08.609 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:08.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.637+0000 7f7502a4d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T14:21:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.637+0000 7f7502a4d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T14:21:08.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.677+0000 7f7502a4d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:08.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.681+0000 7f7502a4d540 -1 journal FileJournal::_open_file : zeroing journal 2024-09-18T14:21:08.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.937+0000 7f7502a4d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:08.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.937+0000 7f7502a4d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:08.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:08.937+0000 7f7502a4d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:08.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T14:21:09.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:09.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:09.077+0000 7fe3cea0b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T14:21:09.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:09.081+0000 7fe3cea0b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T14:21:09.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:09.105+0000 7fe3cea0b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:09.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:09.105+0000 7fe3cea0b540 -1 journal FileJournal::_open_file : zeroing journal 2024-09-18T14:21:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:09.349+0000 7fe3cea0b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:09.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:09.353+0000 7fe3cea0b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:09.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:09.353+0000 7fe3cea0b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:09.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T14:21:09.425 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-18T14:21:09.442 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:09.442 INFO:tasks.ceph:role: osd.3 2024-09-18T14:21:09.442 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:21:09.442 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T14:21:10.003 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:21:10.004 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:21:10.005 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-18T14:21:10.005 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-18T14:21:10.080 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-18T14:21:10.093 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:21:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:21:10.094 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-18T14:21:10.104 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:10.104 INFO:tasks.ceph:role: osd.4 2024-09-18T14:21:10.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:21:10.104 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T14:21:10.624 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:21:10.625 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:21:10.626 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:21:10.627 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-18T14:21:10.627 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-18T14:21:10.698 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-18T14:21:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:21:10.708 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:21:10.709 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-18T14:21:10.723 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T14:21:10.724 INFO:tasks.ceph:role: osd.5 2024-09-18T14:21:10.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:21:10.724 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:21:11.180 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:21:11.181 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:21:11.181 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-18T14:21:11.182 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-18T14:21:11.227 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-18T14:21:11.238 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:21:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:21:11.239 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:11.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.307+0000 7f86742b3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-18T14:21:11.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.307+0000 7f86742b3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-18T14:21:11.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.323+0000 7f86742b3540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:11.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.327+0000 7f86742b3540 -1 journal FileJournal::_open_file : zeroing journal 2024-09-18T14:21:11.555 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.547+0000 7f86742b3540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:11.558 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.551+0000 7f86742b3540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:11.558 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.551+0000 7f86742b3540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:11.581 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-18T14:21:11.596 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:11.684 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.675+0000 7f911485b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-18T14:21:11.685 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.679+0000 7f911485b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-18T14:21:11.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.711+0000 7f911485b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:11.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.715+0000 7f911485b540 -1 journal FileJournal::_open_file : zeroing journal 2024-09-18T14:21:11.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.935+0000 7f911485b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:11.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.935+0000 7f911485b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:11.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:11.935+0000 7f911485b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:11.967 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T14:21:11.985 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:12.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:12.019+0000 7f22ed5db540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-18T14:21:12.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:12.023+0000 7f22ed5db540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-18T14:21:12.060 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:12.051+0000 7f22ed5db540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:12.062 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:12.055+0000 7f22ed5db540 -1 journal FileJournal::_open_file : zeroing journal 2024-09-18T14:21:12.281 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:12.275+0000 7f22ed5db540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:12.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:12.275+0000 7f22ed5db540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:12.282 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:12.275+0000 7f22ed5db540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:21:12.305 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T14:21:12.365 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T14:21:12.365 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:12.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T14:21:12.381 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:12.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T14:21:12.393 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:12.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T14:21:12.403 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:12.404 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T14:21:12.420 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:12.420 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-18T14:21:12.435 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:12.435 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-18T14:21:12.494 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:12.494 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-18T14:21:12.510 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:12.510 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-18T14:21:12.515 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T14:21:12.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T14:21:12.517 DEBUG:teuthology.orchestra.run.smithi174:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDU4epmvrxKDxAAqEp0vX1qEe9+lI5LSgTgwQ== 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDU4epmp1EpJhAAGtkYyj7wNIOvp1noYqdZyQ== 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDV4epm7YX2BBAAMeQqtJNk5rgDXfVwFx6gYA== 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.x] 2024-09-18T14:21:12.529 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDS4epmUBJ+CRAAOt6hORIU63qknGG4dj/aKQ== 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.3] 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDX4epmPxmJEhAAhp7cs8Mx1Gwe1qoNCzFpDA== 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.4] 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDX4epmqnOpKBAAJSqf72mZC3Yk4SBkOC13uQ== 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.5] 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDY4epmuXuGARAAdFelmlUSiUjvPNFRulp0Zw== 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2024-09-18T14:21:12.530 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDS4epmjxmnCxAAdQeKTJ7jsgZucGW5qRhgpw== 2024-09-18T14:21:12.532 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.0] 2024-09-18T14:21:12.532 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDU4epmvrxKDxAAqEp0vX1qEe9+lI5LSgTgwQ== 2024-09-18T14:21:12.532 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.1] 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDU4epmp1EpJhAAGtkYyj7wNIOvp1noYqdZyQ== 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.2] 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDV4epm7YX2BBAAMeQqtJNk5rgDXfVwFx6gYA== 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.x] 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDS4epmUBJ+CRAAOt6hORIU63qknGG4dj/aKQ== 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.3] 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDX4epmPxmJEhAAhp7cs8Mx1Gwe1qoNCzFpDA== 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.4] 2024-09-18T14:21:12.533 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDX4epmqnOpKBAAJSqf72mZC3Yk4SBkOC13uQ== 2024-09-18T14:21:12.534 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.5] 2024-09-18T14:21:12.534 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDY4epmuXuGARAAdFelmlUSiUjvPNFRulp0Zw== 2024-09-18T14:21:12.534 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-09-18T14:21:12.534 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQDS4epmjxmnCxAAdQeKTJ7jsgZucGW5qRhgpw== 2024-09-18T14:21:12.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.536 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.622 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.669 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T14:21:12.756 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T14:21:12.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.847 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.893 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.976 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:12.978 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:21:13.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T14:21:13.025 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T14:21:13.068 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T14:21:13.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T14:21:13.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T14:21:13.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T14:21:13.146 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T14:21:13.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T14:21:13.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T14:21:13.269 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T14:21:13.286 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T14:21:13.339 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T14:21:13.351 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:13.353 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:21:13.359 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T14:21:13.360 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T14:21:13.360 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T14:21:13.403 INFO:tasks.ceph.mon.a:Started 2024-09-18T14:21:13.404 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T14:21:13.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T14:21:13.406 INFO:tasks.ceph.mon.c:Started 2024-09-18T14:21:13.406 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T14:21:13.406 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T14:21:13.408 INFO:tasks.ceph.mon.b:Started 2024-09-18T14:21:13.408 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T14:21:13.409 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T14:21:13.409 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T14:21:13.411 INFO:tasks.ceph.mgr.x:Started 2024-09-18T14:21:13.412 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T14:21:13.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T14:21:13.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.520+0000 7f76e22bf700 10 monclient: get_monmap_and_config 2024-09-18T14:21:13.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.520+0000 7f76e22bf700 10 monclient: build_initial_monmap 2024-09-18T14:21:13.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.520+0000 7f76e22bf700 10 monclient: monmap: 2024-09-18T14:21:13.529 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:13.529 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:13.529 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:13.526513+0000 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:13.526513+0000 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:13.530 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:13.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.520+0000 7f76e22bf700 1 Processor -- start 2024-09-18T14:21:13.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.520+0000 7f76e22bf700 1 -- start start 2024-09-18T14:21:13.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.520+0000 7f76e22bf700 10 monclient: init 2024-09-18T14:21:13.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient(hunting): picked mon.noname-c con 0x7f76dc0a1aa0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient(hunting): picked mon.noname-a con 0x7f76dc0986d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76dc09c660 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc094f20 con 0x7f76dc09c660 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76dc0996f0 con 0x7f76dc0986d0 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76dc0990a0 con 0x7f76dc0a1aa0 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e22bf700 10 monclient(hunting): authenticate will time out at 794.516271s 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34346/0 (socket says 172.21.15.104:34346) 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 learned_addr learned my addr 172.21.15.104:0/1662812012 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76dc0986d0 auth_method 0 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 10 monclient(hunting): get_auth_request con 0x7f76dc0a1aa0 auth_method 0 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.524+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 >> v1:172.21.15.174:6789/0 conn(0x7f76dc09c660 legacy=0x7f76dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:13.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:13.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76dc0986d0 auth_method 0 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 10 monclient(hunting): get_auth_request con 0x7f76dc0a1aa0 auth_method 0 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:21:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:13.728+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:21:14.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:14.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76dc0986d0 auth_method 0 2024-09-18T14:21:14.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:14.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:14.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 10 monclient(hunting): get_auth_request con 0x7f76dc0a1aa0 auth_method 0 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:14.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:14.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.132+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:21:14.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:14.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76dc0986d0 auth_method 0 2024-09-18T14:21:14.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 10 monclient(hunting): get_auth_request con 0x7f76dc0a1aa0 auth_method 0 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:14.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:14.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:21:14.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:14.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:14.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:14.936+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:21:16.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 10 monclient(hunting): tick 2024-09-18T14:21:16.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 monclient(hunting): continuing hunt 2024-09-18T14:21:16.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T14:21:16.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 msgr2=0x7f76dc09bbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:16.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76dc0a1aa0 0x7f76dc09bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:16.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 msgr2=0x7f76dc09c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:16.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76dc0986d0 0x7f76dc09c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:16.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 >> v1:172.21.15.174:6789/0 conn(0x7f76dc09c660 legacy=0x7f76dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:16.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 10 monclient: _add_conns ranks=[0,2] 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76c8003760 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f76c8007420 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc098dc0 con 0x7f76c8003760 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc0996f0 con 0x7f76c8007420 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76d9ffb700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76c8007420 auth_method 0 2024-09-18T14:21:16.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:16.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:16.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:16.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:16.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:16.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:16.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:16.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.524+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v1:172.21.15.104:6789/0 conn(0x7f76c8003760 legacy=0x7f76c8005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:16.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:16.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76c8007420 auth_method 0 2024-09-18T14:21:16.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:16.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:16.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:16.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:16.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:16.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:16.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:16.728+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:21:17.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:17.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76c8007420 auth_method 0 2024-09-18T14:21:17.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:17.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:17.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:17.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:17.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:17.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:17.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.132+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76c8007420 auth_method 0 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:17.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:17.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:17.936+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:21:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): tick 2024-09-18T14:21:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 monclient(hunting): continuing hunt 2024-09-18T14:21:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T14:21:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 msgr2=0x7f76c80098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c8007420 0x7f76c80098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 >> v1:172.21.15.104:6789/0 conn(0x7f76c8003760 legacy=0x7f76c8005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 reap_dead start 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76c800a160 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 0x7f76c8005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f76c80098b0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 0x7f76c800d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f76c800b380 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c800b090 con 0x7f76c800a160 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76dc0996f0 con 0x7f76c80098b0 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 1 -- 172.21.15.104:0/1662812012 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76dc0990a0 con 0x7f76c800b380 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76d9ffb700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 0x7f76c8005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76c80098b0 auth_method 0 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 0x7f76c800d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 msgr2=0x7f76c8005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 10 monclient(hunting): get_auth_request con 0x7f76c800b380 auth_method 0 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 msgr2=0x7f76c8005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 0x7f76c8005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 0x7f76c8005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 msgr2=0x7f76c800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 msgr2=0x7f76c800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 0x7f76c800d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76e085c700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 0x7f76c800d760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:19.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.524+0000 7f76dbfff700 1 -- 172.21.15.104:0/1662812012 >> v1:172.21.15.104:6789/0 conn(0x7f76c800a160 legacy=0x7f76c800a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:19.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738250190 0 0) 0x7f76c800b090 con 0x7f76c800a160 2024-09-18T14:21:19.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): my global_id is 4098 2024-09-18T14:21:19.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c4003570 con 0x7f76c800a160 2024-09-18T14:21:19.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1374050840 0 0) 0x7f76c4003570 con 0x7f76c800a160 2024-09-18T14:21:19.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 msgr2=0x7f76c800d760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:19.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.174:3300/0 conn(0x7f76c800b380 0x7f76c800d760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 msgr2=0x7f76c8005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 --2- 172.21.15.104:0/1662812012 >> v2:172.21.15.104:3300/0 conn(0x7f76c80098b0 0x7f76c8005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: _finish_hunting 0 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 monclient: found mon.noname-a-legacy 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.104:6789/0 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c800ecb0 con 0x7f76c800a160 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:19.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e22bf700 5 monclient: authenticate success, global_id 4098 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7f76cc00d4c0 con 0x7f76c800a160 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e22bf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: dump: 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.555 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 >> v1:172.21.15.104:6789/0 conn(0x7f76c800a160 legacy=0x7f76c800a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76c4003ab0 0x7f76c4003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): picked mon.b con 0x7f76c4003ab0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c4004810 0x7f76c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): picked mon.c con 0x7f76c4004810 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c40052e0 0x7f76c400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): picked mon.a con 0x7f76c40052e0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc0996f0 con 0x7f76c40052e0 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc0990a0 con 0x7f76c4003ab0 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c400c2b0 con 0x7f76c4004810 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76daffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e22bf700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c40052e0 0x7f76c400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76c4003ab0 0x7f76c4003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 10 monclient(hunting): get_auth_request con 0x7f76c40052e0 auth_method 0 2024-09-18T14:21:19.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 10 monclient(hunting): get_auth_request con 0x7f76c4003ab0 auth_method 0 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c4004810 0x7f76c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76e085c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76dbfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76c4004810 auth_method 0 2024-09-18T14:21:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.540+0000 7f76db7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c4004810 msgr2=0x7f76c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c4004810 0x7f76c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76c4003ab0 msgr2=0x7f76c4003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76c4003ab0 0x7f76c4003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 monclient: found mon.a 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c4003570 con 0x7f76c40052e0 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 10 monclient: _finish_auth 0 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76c4003ab0 0x7f76c4003ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 reap_dead start 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c40052e0 msgr2=0x7f76c400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e085c700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c40052e0 0x7f76c400ba60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76dc069b80 tx=0x7f76d0007ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472d2852aeb12ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0007070 con 0x7f76c40052e0 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 10 monclient: dump: 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.560 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d00071d0 con 0x7f76c40052e0 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e22bf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e22bf700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 1 -- 172.21.15.104:0/1662812012 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0007570 con 0x7f76c40052e0 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76e22bf700 4 set_mon_vals no callback set 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.544+0000 7f76daffd700 10 monclient: dump: 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.562 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:19.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient: shutdown 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c40052e0 msgr2=0x7f76c400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c40052e0 0x7f76c400ba60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76dc069b80 tx=0x7f76d0007ba0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- 172.21.15.104:0/1662812012 shutdown_connections 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 --2- 172.21.15.104:0/1662812012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c40052e0 0x7f76c400ba60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- 172.21.15.104:0/1662812012 >> 172.21.15.104:0/1662812012 conn(0x7f76dc069540 msgr2=0x7f76dc0a11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- 172.21.15.104:0/1662812012 shutdown_connections 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- 172.21.15.104:0/1662812012 wait complete. 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient: build_initial_monmap 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient: monmap: 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 Processor -- start 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- start start 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient: init 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:19.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c40052e0 0x7f76dc1416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): picked mon.noname-c con 0x7f76c40052e0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76dc091960 0x7f76dc141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): picked mon.noname-a con 0x7f76dc091960 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76dc08d180 0x7f76dc08d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): picked mon.noname-b con 0x7f76dc08d180 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc0a0080 con 0x7f76dc091960 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc090a20 con 0x7f76dc08d180 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc08f8e0 con 0x7f76c40052e0 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e22bf700 10 monclient(hunting): authenticate will time out at 800.543463s 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76dc091960 0x7f76dc141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c40052e0 0x7f76dc1416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76dc091960 0x7f76dc141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34400/0 (socket says 172.21.15.104:34400) 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 1 -- 172.21.15.104:0/1739924530 learned_addr learned my addr 172.21.15.104:0/1739924530 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 10 monclient(hunting): get_auth_request con 0x7f76dc091960 auth_method 0 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 10 monclient(hunting): get_auth_request con 0x7f76c40052e0 auth_method 0 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76db7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e085c700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76dc08d180 0x7f76dc08d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76dbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e085c700 10 monclient(hunting): get_auth_request con 0x7f76dc08d180 auth_method 0 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e085c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e085c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.548+0000 7f76e085c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e085c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e085c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e085c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:19.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 -- 172.21.15.104:0/1739924530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76dc08d180 msgr2=0x7f76dc08d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76dc08d180 0x7f76dc08d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 -- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76dc091960 msgr2=0x7f76dc141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76dc091960 0x7f76dc141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 monclient: found mon.noname-c 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 -- 172.21.15.104:0/1739924530 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc090740 con 0x7f76c40052e0 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76db7fe700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76dc091960 0x7f76dc141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 -- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c40052e0 msgr2=0x7f76dc1416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76dbfff700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c40052e0 0x7f76dc1416a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f76cc001eb0 tx=0x7f76cc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 1 -- 172.21.15.104:0/1739924530 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc014a90 con 0x7f76c40052e0 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: dump: 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e22bf700 5 monclient: authenticate success, global_id 4100 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 1 -- 172.21.15.104:0/1739924530 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc014bf0 con 0x7f76c40052e0 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 1 -- 172.21.15.104:0/1739924530 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc014da0 con 0x7f76c40052e0 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e22bf700 10 monclient: _renew_subs 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e22bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc090d00 con 0x7f76c40052e0 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76da7fc700 4 set_mon_vals no callback set 2024-09-18T14:21:19.575 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 10 monclient: dump: 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e22bf700 10 monclient: _renew_subs 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e22bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:19.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc0942c0 con 0x7f76c40052e0 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 1 -- 172.21.15.104:0/1739924530 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f76cc01d020 con 0x7f76c40052e0 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.552+0000 7f76c3fff700 1 -- 172.21.15.104:0/1739924530 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f76cc020770 con 0x7f76c40052e0 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.556+0000 7f76e22bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.556+0000 7f76e22bf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.556+0000 7f76e22bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.556+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc064b30 con 0x7f76c40052e0 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.560+0000 7f76c3fff700 1 -- 172.21.15.104:0/1739924530 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f76cc025060 con 0x7f76c40052e0 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.560+0000 7f76c3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:19.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.560+0000 7f76c3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:19.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.628+0000 7f76e22bf700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T14:21:19.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.628+0000 7f76e22bf700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T14:21:19.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.628+0000 7f76e22bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:19.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.628+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f76dc068c80 con 0x7f76c40052e0 2024-09-18T14:21:19.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76c3fff700 1 -- 172.21.15.104:0/1739924530 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f76cc017650 con 0x7f76c40052e0 2024-09-18T14:21:19.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76c3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T14:21:19.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76c3fff700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 10 monclient: shutdown 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c40052e0 msgr2=0x7f76dc1416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c40052e0 0x7f76dc1416a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f76cc001eb0 tx=0x7f76cc008040 comp rx=0 tx=0).stop 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 shutdown_connections 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76dc08d180 0x7f76dc08d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76dc091960 0x7f76dc141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 --2- 172.21.15.104:0/1739924530 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f76c40052e0 0x7f76dc1416a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 >> 172.21.15.104:0/1739924530 conn(0x7f76dc069540 msgr2=0x7f76dc0a0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 shutdown_connections 2024-09-18T14:21:19.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:19.640+0000 7f76e22bf700 1 -- 172.21.15.104:0/1739924530 wait complete. 2024-09-18T14:21:19.651 INFO:teuthology.orchestra.run.smithi104.stderr:adjusted tunables profile to default 2024-09-18T14:21:19.665 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T14:21:19.665 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:19.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T14:21:19.678 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:19.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T14:21:19.738 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-18T14:21:19.738 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T14:21:19.747 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:19.748 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-18T14:21:19.763 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:19.763 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-18T14:21:19.820 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T14:21:19.820 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-18T14:21:19.830 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new f9ec9d12-b31d-49ef-a56d-c7fa21e03aad 0 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.935+0000 7f261ee8c700 10 monclient: get_monmap_and_config 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.935+0000 7f261ee8c700 10 monclient: build_initial_monmap 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.935+0000 7f261ee8c700 10 monclient: monmap: 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:19.941070+0000 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:19.941070+0000 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:19.944 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:19.945 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:19.945 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:19.945 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:19.945 INFO:teuthology.orchestra.run.smithi174.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:19.945 INFO:teuthology.orchestra.run.smithi174.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:19.945 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.935+0000 7f261ee8c700 1 Processor -- start 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 1 -- start start 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient: init 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f261809a6b0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f261809aa80 0x7f26181322f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient(hunting): picked mon.noname-a con 0x7f261809aa80 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2618099e60 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261809b9c0 con 0x7f261809a6b0 2024-09-18T14:21:19.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26180a1280 con 0x7f2618099e60 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f261809b6e0 con 0x7f261809aa80 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ee8c700 10 monclient(hunting): authenticate will time out at 798.189092s 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261e68b700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f2618099e60 0x7f2618138f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59156/0 (socket says 172.21.15.174:59156) 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261e68b700 1 -- 172.21.15.174:0/1279399647 learned_addr learned my addr 172.21.15.174:0/1279399647 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 1 --2- 172.21.15.174:0/1279399647 >> v2:172.21.15.104:3300/0 conn(0x7f261809aa80 0x7f26181322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 10 monclient(hunting): get_auth_request con 0x7f261809aa80 auth_method 0 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261e68b700 1 -- 172.21.15.174:0/1279399647 >> v1:172.21.15.174:6789/0 conn(0x7f2618099e60 legacy=0x7f2618138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210510111 0 0) 0x7f26180a1280 con 0x7f2618099e60 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ce88700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ce88700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ce88700 10 monclient(hunting): my global_id is 4105 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.939+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2604003570 con 0x7f2618099e60 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 1 -- 172.21.15.174:0/1279399647 >> v1:172.21.15.104:6789/0 conn(0x7f261809a6b0 legacy=0x7f2618131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:19.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2718527112 0 0) 0x7f2604003570 con 0x7f2618099e60 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 >> v2:172.21.15.104:3300/0 conn(0x7f261809aa80 msgr2=0x7f26181322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 --2- 172.21.15.174:0/1279399647 >> v2:172.21.15.104:3300/0 conn(0x7f261809aa80 0x7f26181322f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 >> v1:172.21.15.104:6789/0 conn(0x7f261809a6b0 legacy=0x7f2618131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: _finish_hunting 0 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 monclient: found mon.noname-c-legacy 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.174:6789/0 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2618132830 con 0x7f2618099e60 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: _finish_auth 0 2024-09-18T14:21:19.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7f2610001b20 con 0x7f2618099e60 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ee8c700 5 monclient: authenticate success, global_id 4105 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ee8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 --2- 172.21.15.174:0/1279399647 >> v2:172.21.15.104:3300/0 conn(0x7f261809aa80 0x7f26181322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: dump: 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.956 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] but i'm connected to v1:172.21.15.174:6789/0, reconnecting 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 >> v1:172.21.15.174:6789/0 conn(0x7f2618099e60 legacy=0x7f2618138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604003c50 0x7f2604004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): picked mon.b con 0x7f2604003c50 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2604004810 0x7f2604004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): picked mon.c con 0x7f2604004810 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26040053c0 0x7f260400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): picked mon.a con 0x7f26040053c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261809b6e0 con 0x7f26040053c0 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260400c360 con 0x7f2604003c50 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260400c520 con 0x7f2604004810 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ce88700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261ee8c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604003c50 0x7f2604004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2604004810 0x7f2604004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26040053c0 0x7f260400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.958 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): get_auth_request con 0x7f2604004810 auth_method 0 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 10 monclient(hunting): get_auth_request con 0x7f2604003c50 auth_method 0 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 10 monclient(hunting): get_auth_request con 0x7f26040053c0 auth_method 0 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261de8a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 -- 172.21.15.174:0/1279399647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604003c50 msgr2=0x7f2604004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604003c50 0x7f2604004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 -- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26040053c0 msgr2=0x7f260400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:19.960 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26040053c0 0x7f260400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient: _finish_hunting 0 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 monclient: found mon.c 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 -- 172.21.15.174:0/1279399647 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2604003570 con 0x7f2604004810 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261e68b700 1 -- 172.21.15.174:0/1279399647 reap_dead start 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 10 monclient: _finish_auth 0 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 -- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2604004810 msgr2=0x7f2604004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.943+0000 7f261d689700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2604004810 0x7f2604004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2614009df0 tx=0x7f26140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f8cc74008a9b74a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614009740 con 0x7f2604004810 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:19.961 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 10 monclient: dump: 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261400eb30 con 0x7f2604004810 2024-09-18T14:21:19.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 4 set_mon_vals no callback set 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 1 -- 172.21.15.174:0/1279399647 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614013470 con 0x7f2604004810 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ce88700 10 monclient: dump: 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.963 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 10 monclient: shutdown 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 -- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2604004810 msgr2=0x7f2604004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2604004810 0x7f2604004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2614009df0 tx=0x7f26140091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 -- 172.21.15.174:0/1279399647 shutdown_connections 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 --2- 172.21.15.174:0/1279399647 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2604004810 0x7f2604004c80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 -- 172.21.15.174:0/1279399647 >> 172.21.15.174:0/1279399647 conn(0x7f26180a53b0 msgr2=0x7f26180a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 -- 172.21.15.174:0/1279399647 shutdown_connections 2024-09-18T14:21:19.964 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 -- 172.21.15.174:0/1279399647 wait complete. 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 10 monclient: build_initial_monmap 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 10 monclient: monmap: 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:19.969 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 Processor -- start 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 1 -- start start 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.947+0000 7f261ee8c700 10 monclient: init 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 0x7f2618130da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): picked mon.noname-b con 0x7f2604004810 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f261811e070 0x7f2618141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): picked mon.noname-a con 0x7f261811e070 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:19.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f26181312e0 0x7f2618141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): picked mon.noname-c con 0x7f26181312e0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2618142060 con 0x7f261811e070 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261811f1b0 con 0x7f2604004810 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261811eed0 con 0x7f26181312e0 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient(hunting): authenticate will time out at 798.200189s 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 0x7f2618130da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 0x7f2618130da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43188/0 (socket says 172.21.15.174:43188) 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 -- 172.21.15.174:0/2525415512 learned_addr learned my addr 172.21.15.174:0/2525415512 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-18T14:21:19.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): get_auth_request con 0x7f2604004810 auth_method 0 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f261811e070 0x7f2618141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f26181312e0 0x7f2618141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 10 monclient(hunting): get_auth_request con 0x7f261811e070 auth_method 0 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 10 monclient(hunting): get_auth_request con 0x7f26181312e0 auth_method 0 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261e68b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261d689700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 -- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f26181312e0 msgr2=0x7f2618141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f26181312e0 0x7f2618141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 -- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f261811e070 msgr2=0x7f2618141140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f261811e070 0x7f2618141140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient: _finish_hunting 0 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 monclient: found mon.noname-b 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 -- 172.21.15.174:0/2525415512 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261811f490 con 0x7f2604004810 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 -- 172.21.15.174:0/2525415512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 msgr2=0x7f2618130da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261de8a700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 0x7f2618130da0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2608001670 tx=0x7f2608007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 1 -- 172.21.15.174:0/2525415512 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2608005c30 con 0x7f2604004810 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 10 monclient: dump: 2024-09-18T14:21:19.974 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.975 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 5 monclient: authenticate success, global_id 4108 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 1 -- 172.21.15.174:0/2525415512 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2608005d90 con 0x7f2604004810 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient: _renew_subs 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261811ba10 con 0x7f2604004810 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.951+0000 7f260dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f260dffb700 1 -- 172.21.15.174:0/2525415512 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2608006260 con 0x7f2604004810 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f260dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f260dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f260f7fe700 4 set_mon_vals no callback set 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f260dffb700 10 monclient: dump: 2024-09-18T14:21:19.976 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f261ee8c700 10 monclient: _renew_subs 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f261ee8c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261811bf30 con 0x7f2604004810 2024-09-18T14:21:19.977 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f260dffb700 1 -- 172.21.15.174:0/2525415512 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f26080065d0 con 0x7f2604004810 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f260dffb700 1 -- 172.21.15.174:0/2525415512 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2608014e30 con 0x7f2604004810 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f261ee8c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f261ee8c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f261ee8c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.955+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2618063340 con 0x7f2604004810 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.959+0000 7f260dffb700 1 -- 172.21.15.174:0/2525415512 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f2608023060 con 0x7f2604004810 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.959+0000 7f260dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:19.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:19.959+0000 7f260dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:20.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.043+0000 7f261ee8c700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f9ec9d12-b31d-49ef-a56d-c7fa21e03aad", "id": 0}] 2024-09-18T14:21:20.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.043+0000 7f261ee8c700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f9ec9d12-b31d-49ef-a56d-c7fa21e03aad", "id": 0}] 2024-09-18T14:21:20.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.043+0000 7f261ee8c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.043+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f9ec9d12-b31d-49ef-a56d-c7fa21e03aad", "id": 0} v 0) v1 -- 0x7f26180689e0 con 0x7f2604004810 2024-09-18T14:21:20.062 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f260dffb700 1 -- 172.21.15.174:0/2525415512 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f9ec9d12-b31d-49ef-a56d-c7fa21e03aad", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2608019070 con 0x7f2604004810 2024-09-18T14:21:20.062 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f260dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f9ec9d12-b31d-49ef-a56d-c7fa21e03aad", "id": 0}] 2024-09-18T14:21:20.062 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f260dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:20.063 INFO:teuthology.orchestra.run.smithi174.stdout:0 2024-09-18T14:21:20.065 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 10 monclient: shutdown 2024-09-18T14:21:20.065 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.065 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 msgr2=0x7f2618130da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.065 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 0x7f2618130da0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2608001670 tx=0x7f2608007d10 comp rx=0 tx=0).stop 2024-09-18T14:21:20.065 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 shutdown_connections 2024-09-18T14:21:20.066 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f26181312e0 0x7f2618141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.066 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f261811e070 0x7f2618141140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.066 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 --2- 172.21.15.174:0/2525415512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2604004810 0x7f2618130da0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.066 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 >> 172.21.15.174:0/2525415512 conn(0x7f26180a53b0 msgr2=0x7f261809cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.066 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 shutdown_connections 2024-09-18T14:21:20.066 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.055+0000 7f261ee8c700 1 -- 172.21.15.174:0/2525415512 wait complete. 2024-09-18T14:21:20.077 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 9522965d-f7f2-4d3e-8c54-e2377d2bfb56 1 2024-09-18T14:21:20.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient: get_monmap_and_config 2024-09-18T14:21:20.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient: monmap: 2024-09-18T14:21:20.196 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.197 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.197 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:20.178674+0000 2024-09-18T14:21:20.197 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:20.178674+0000 2024-09-18T14:21:20.197 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.197 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.197 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:20.198 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:20.198 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:20.198 INFO:teuthology.orchestra.run.smithi174.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:20.198 INFO:teuthology.orchestra.run.smithi174.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:20.198 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 1 Processor -- start 2024-09-18T14:21:20.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 1 -- start start 2024-09-18T14:21:20.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient: init 2024-09-18T14:21:20.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:20.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T14:21:20.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f28f009a6b0 0x7f28f0131be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient(hunting): picked mon.noname-c con 0x7f28f009a6b0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:20.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28f009aa80 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:20.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f28f0099e60 0x7f28f0138f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient(hunting): picked mon.noname-a con 0x7f28f0099e60 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:20.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f009b9c0 con 0x7f28f009aa80 2024-09-18T14:21:20.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f00a1280 con 0x7f28f0099e60 2024-09-18T14:21:20.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f009b6e0 con 0x7f28f009a6b0 2024-09-18T14:21:20.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:20.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f7429700 10 monclient(hunting): authenticate will time out at 798.425286s 2024-09-18T14:21:20.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f28f009a6b0 0x7f28f0131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.202 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f5c26700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f28f009aa80 0x7f28f0132120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59172/0 (socket says 172.21.15.174:59172) 2024-09-18T14:21:20.202 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f5c26700 1 -- 172.21.15.174:0/2488025986 learned_addr learned my addr 172.21.15.174:0/2488025986 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-09-18T14:21:20.202 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): get_auth_request con 0x7f28f009a6b0 auth_method 0 2024-09-18T14:21:20.202 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.203 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.203 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.203 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 1 --2- 172.21.15.174:0/2488025986 >> v2:172.21.15.104:3300/0 conn(0x7f28f0099e60 0x7f28f0138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.203 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.203 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.203 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.203 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 10 monclient(hunting): get_auth_request con 0x7f28f0099e60 auth_method 0 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6c28700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-09-18T14:21:20.204 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 1 -- 172.21.15.174:0/2488025986 >> v2:172.21.15.104:3300/0 conn(0x7f28f0099e60 msgr2=0x7f28f0138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 1 --2- 172.21.15.174:0/2488025986 >> v2:172.21.15.104:3300/0 conn(0x7f28f0099e60 0x7f28f0138f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 1 -- 172.21.15.174:0/2488025986 >> v1:172.21.15.174:6789/0 conn(0x7f28f009aa80 legacy=0x7f28f0132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient: _finish_hunting 0 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 1 monclient: found mon.noname-c 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.175+0000 7f28f6427700 1 -- 172.21.15.174:0/2488025986 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f0132830 con 0x7f28f009a6b0 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 1 -- 172.21.15.174:0/2488025986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f009a6b0 msgr2=0x7f28f0131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 1 --2- 172.21.15.174:0/2488025986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f009a6b0 0x7f28f0131be0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f28f00a63e0 tx=0x7f28e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16cfa158d79587c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 1 -- 172.21.15.174:0/2488025986 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4011840 con 0x7f28f009a6b0 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: dump: 2024-09-18T14:21:20.205 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: _finish_auth 0 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 1 -- 172.21.15.174:0/2488025986 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e40119a0 con 0x7f28f009a6b0 2024-09-18T14:21:20.206 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 1 -- 172.21.15.174:0/2488025986 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4011d40 con 0x7f28f009a6b0 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5425700 10 monclient: dump: 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.207 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 5 monclient: authenticate success, global_id 4117 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 4 set_mon_vals no callback set 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient: shutdown 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- 172.21.15.174:0/2488025986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f009a6b0 msgr2=0x7f28f0131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 --2- 172.21.15.174:0/2488025986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f009a6b0 0x7f28f0131be0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f28f00a63e0 tx=0x7f28e40091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- 172.21.15.174:0/2488025986 shutdown_connections 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 --2- 172.21.15.174:0/2488025986 >> v2:172.21.15.104:3300/0 conn(0x7f28f0099e60 0x7f28f0138f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 --2- 172.21.15.174:0/2488025986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f009a6b0 0x7f28f0131be0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- 172.21.15.174:0/2488025986 >> 172.21.15.174:0/2488025986 conn(0x7f28f00a53b0 msgr2=0x7f28f00a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- 172.21.15.174:0/2488025986 shutdown_connections 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- 172.21.15.174:0/2488025986 wait complete. 2024-09-18T14:21:20.208 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient: monmap: 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 Processor -- start 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- start start 2024-09-18T14:21:20.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient: init 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 0x7f28f0134610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): picked mon.noname-b con 0x7f28f0099e60 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f28f009a6b0 0x7f28f0134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): picked mon.noname-c con 0x7f28f009a6b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28f01389d0 0x7f28f0135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): picked mon.noname-a con 0x7f28f01389d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00a3d30 con 0x7f28f01389d0 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0137f10 con 0x7f28f0099e60 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.210 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f009ae60 con 0x7f28f009a6b0 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f7429700 10 monclient(hunting): authenticate will time out at 798.431127s 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 0x7f28f0134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 0x7f28f0134610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43238/0 (socket says 172.21.15.174:43238) 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 1 -- 172.21.15.174:0/385258252 learned_addr learned my addr 172.21.15.174:0/385258252 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 10 monclient(hunting): get_auth_request con 0x7f28f0099e60 auth_method 0 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28f01389d0 0x7f28f0135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f28f009a6b0 0x7f28f0134b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6427700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 10 monclient(hunting): get_auth_request con 0x7f28f01389d0 auth_method 0 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 10 monclient(hunting): get_auth_request con 0x7f28f009a6b0 auth_method 0 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f6c28700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.179+0000 7f28f5c26700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 -- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f28f009a6b0 msgr2=0x7f28f0134b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f28f009a6b0 0x7f28f0134b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 -- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28f01389d0 msgr2=0x7f28f0135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28f01389d0 0x7f28f0135090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 10 monclient: _finish_hunting 0 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 monclient: found mon.noname-b 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 -- 172.21.15.174:0/385258252 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4009420 con 0x7f28f0099e60 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 -- 172.21.15.174:0/385258252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 msgr2=0x7f28f0134610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f6427700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 0x7f28f0134610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f28e40091e0 tx=0x7f28e4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:20.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 1 -- 172.21.15.174:0/385258252 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e400bce0 con 0x7f28f0099e60 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: dump: 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 1 -- 172.21.15.174:0/385258252 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e400be40 con 0x7f28f0099e60 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f7429700 5 monclient: authenticate success, global_id 4120 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f7429700 10 monclient: _renew_subs 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f7429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f0137c30 con 0x7f28f0099e60 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 1 -- 172.21.15.174:0/385258252 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e401a6b0 con 0x7f28f0099e60 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 10 monclient: dump: 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.215 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e3fff700 4 set_mon_vals no callback set 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f7429700 10 monclient: _renew_subs 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f7429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f0136cd0 con 0x7f28f0099e60 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 1 -- 172.21.15.174:0/385258252 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f28e401aae0 con 0x7f28f0099e60 2024-09-18T14:21:20.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.183+0000 7f28e27fc700 1 -- 172.21.15.174:0/385258252 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f28e4025ad0 con 0x7f28f0099e60 2024-09-18T14:21:20.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.187+0000 7f28f7429700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.187+0000 7f28f7429700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.187+0000 7f28f7429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.187+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f0063340 con 0x7f28f0099e60 2024-09-18T14:21:20.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.191+0000 7f28e27fc700 1 -- 172.21.15.174:0/385258252 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f28e402a020 con 0x7f28f0099e60 2024-09-18T14:21:20.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.191+0000 7f28e27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.217 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.191+0000 7f28e27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:20.275 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.267+0000 7f28f7429700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9522965d-f7f2-4d3e-8c54-e2377d2bfb56", "id": 1}] 2024-09-18T14:21:20.275 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.267+0000 7f28f7429700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9522965d-f7f2-4d3e-8c54-e2377d2bfb56", "id": 1}] 2024-09-18T14:21:20.275 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.267+0000 7f28f7429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.275 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.267+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9522965d-f7f2-4d3e-8c54-e2377d2bfb56", "id": 1} v 0) v1 -- 0x7f28f00689e0 con 0x7f28f0099e60 2024-09-18T14:21:20.286 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28e27fc700 1 -- 172.21.15.174:0/385258252 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9522965d-f7f2-4d3e-8c54-e2377d2bfb56", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f28e4025ce0 con 0x7f28f0099e60 2024-09-18T14:21:20.286 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28e27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9522965d-f7f2-4d3e-8c54-e2377d2bfb56", "id": 1}] 2024-09-18T14:21:20.286 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28e27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:20.286 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-09-18T14:21:20.287 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 10 monclient: shutdown 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 msgr2=0x7f28f0134610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 0x7f28f0134610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f28e40091e0 tx=0x7f28e4016000 comp rx=0 tx=0).stop 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 shutdown_connections 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28f01389d0 0x7f28f0135090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f28f009a6b0 0x7f28f0134b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 --2- 172.21.15.174:0/385258252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28f0099e60 0x7f28f0134610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 >> 172.21.15.174:0/385258252 conn(0x7f28f00a53b0 msgr2=0x7f28f009e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 shutdown_connections 2024-09-18T14:21:20.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.279+0000 7f28f7429700 1 -- 172.21.15.174:0/385258252 wait complete. 2024-09-18T14:21:20.301 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 07bd6e83-04fd-43e8-b538-9ee89fdbf1b4 2 2024-09-18T14:21:20.389 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:20.383+0000 7ff37b32f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T14:21:20.405 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient: get_monmap_and_config 2024-09-18T14:21:20.405 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.405 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient: monmap: 2024-09-18T14:21:20.405 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.405 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.405 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:20.402912+0000 2024-09-18T14:21:20.405 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:20.402912+0000 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:20.406 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.412 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 1 Processor -- start 2024-09-18T14:21:20.412 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 1 -- start start 2024-09-18T14:21:20.412 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient: init 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809bab0 0x7fb13806f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient(hunting): picked mon.noname-b con 0x7fb13809bab0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb13809b610 0x7fb13809be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb13809b610 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb13809c9a0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:20.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb138099490 con 0x7fb13809c9a0 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13809a950 con 0x7fb13809b610 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13809af10 con 0x7fb13809bab0 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13759e700 10 monclient(hunting): authenticate will time out at 798.650553s 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb13809b610 0x7fb13809be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb136d9d700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fb13809c9a0 0x7fb13809cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59184/0 (socket says 172.21.15.174:59184) 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb136d9d700 1 -- 172.21.15.174:0/2118830157 learned_addr learned my addr 172.21.15.174:0/2118830157 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-09-18T14:21:20.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): get_auth_request con 0x7fb13809b610 auth_method 0 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13659c700 1 --2- 172.21.15.174:0/2118830157 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809bab0 0x7fb13806f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13659c700 10 monclient(hunting): get_auth_request con 0x7fb13809bab0 auth_method 0 2024-09-18T14:21:20.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 1 -- 172.21.15.174:0/2118830157 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809bab0 msgr2=0x7fb13806f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 1 --2- 172.21.15.174:0/2118830157 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809bab0 0x7fb13806f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 1 -- 172.21.15.174:0/2118830157 >> v1:172.21.15.174:6789/0 conn(0x7fb13809c9a0 legacy=0x7fb13809cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient: _finish_hunting 0 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 1 monclient: found mon.noname-c 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb135d9b700 1 -- 172.21.15.174:0/2118830157 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13809c3c0 con 0x7fb13809b610 2024-09-18T14:21:20.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.399+0000 7fb13659c700 1 --2- 172.21.15.174:0/2118830157 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809bab0 0x7fb13806f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb135d9b700 1 -- 172.21.15.174:0/2118830157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809b610 msgr2=0x7fb13809be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb135d9b700 1 --2- 172.21.15.174:0/2118830157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809b610 0x7fb13809be80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098e0 tx=0x7fb12c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26a638fd3b369b13 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 1 -- 172.21.15.174:0/2118830157 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c009e80 con 0x7fb13809b610 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: dump: 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.417 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: _finish_auth 0 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 5 monclient: authenticate success, global_id 4123 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 1 -- 172.21.15.174:0/2118830157 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c00ba70 con 0x7fb13809b610 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 4 set_mon_vals no callback set 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 1 -- 172.21.15.174:0/2118830157 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c00bdf0 con 0x7fb13809b610 2024-09-18T14:21:20.418 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13559a700 10 monclient: dump: 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 10 monclient: shutdown 2024-09-18T14:21:20.419 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 -- 172.21.15.174:0/2118830157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809b610 msgr2=0x7fb13809be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 --2- 172.21.15.174:0/2118830157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809b610 0x7fb13809be80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098e0 tx=0x7fb12c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 -- 172.21.15.174:0/2118830157 shutdown_connections 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 --2- 172.21.15.174:0/2118830157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809b610 0x7fb13809be80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 --2- 172.21.15.174:0/2118830157 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809bab0 0x7fb13806f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 -- 172.21.15.174:0/2118830157 >> 172.21.15.174:0/2118830157 conn(0x7fb1380a1fb0 msgr2=0x7fb1380a23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 -- 172.21.15.174:0/2118830157 shutdown_connections 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 1 -- 172.21.15.174:0/2118830157 wait complete. 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.403+0000 7fb13759e700 10 monclient: monmap: 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.420 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:20.421 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 Processor -- start 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 -- start start 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient: init 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809b610 0x7fb1380676a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb13809b610 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:20.425 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13809bab0 0x7fb138118ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): picked mon.noname-a con 0x7fb13809bab0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 0x7fb13812ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): picked mon.noname-b con 0x7fb13809c9a0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138119010 con 0x7fb13809bab0 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13806e070 con 0x7fb13809c9a0 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13811f650 con 0x7fb13809b610 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13759e700 10 monclient(hunting): authenticate will time out at 798.657315s 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 0x7fb13812ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.426 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 0x7fb13812ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43252/0 (socket says 172.21.15.174:43252) 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 -- 172.21.15.174:0/359275278 learned_addr learned my addr 172.21.15.174:0/359275278 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): get_auth_request con 0x7fb13809c9a0 auth_method 0 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809b610 0x7fb1380676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 10 monclient(hunting): get_auth_request con 0x7fb13809b610 auth_method 0 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13809bab0 0x7fb138118ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 10 monclient(hunting): get_auth_request con 0x7fb13809bab0 auth_method 0 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb13659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb135d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 -- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809b610 msgr2=0x7fb1380676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809b610 0x7fb1380676a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 -- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13809bab0 msgr2=0x7fb138118ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13809bab0 0x7fb138118ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 monclient: found mon.noname-b 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.407+0000 7fb136d9d700 1 -- 172.21.15.174:0/359275278 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c009420 con 0x7fb13809c9a0 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb136d9d700 1 -- 172.21.15.174:0/359275278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 msgr2=0x7fb13812ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb136d9d700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 0x7fb13812ebb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb13000ccc0 tx=0x7fb130005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 1 -- 172.21.15.174:0/359275278 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000aa90 con 0x7fb13809c9a0 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:20.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: dump: 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:20.430 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 1 -- 172.21.15.174:0/359275278 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13000abf0 con 0x7fb13809c9a0 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb13759e700 5 monclient: authenticate success, global_id 4126 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb13759e700 10 monclient: _renew_subs 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb13759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1381199b0 con 0x7fb13809c9a0 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 1 -- 172.21.15.174:0/359275278 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1300070f0 con 0x7fb13809c9a0 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 10 monclient: dump: 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.431 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb134d99700 4 set_mon_vals no callback set 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb13759e700 10 monclient: _renew_subs 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb13759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13812f0f0 con 0x7fb13809c9a0 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 1 -- 172.21.15.174:0/359275278 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fb1300074e0 con 0x7fb13809c9a0 2024-09-18T14:21:20.432 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.411+0000 7fb1267fc700 1 -- 172.21.15.174:0/359275278 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fb13001b9c0 con 0x7fb13809c9a0 2024-09-18T14:21:20.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.415+0000 7fb13759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.415+0000 7fb13759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.415+0000 7fb13759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.415+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13811f110 con 0x7fb13809c9a0 2024-09-18T14:21:20.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.415+0000 7fb1267fc700 1 -- 172.21.15.174:0/359275278 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7fb130007920 con 0x7fb13809c9a0 2024-09-18T14:21:20.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.415+0000 7fb1267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.433 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.415+0000 7fb1267fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:20.502 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.495+0000 7fb13759e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "07bd6e83-04fd-43e8-b538-9ee89fdbf1b4", "id": 2}] 2024-09-18T14:21:20.502 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.495+0000 7fb13759e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "07bd6e83-04fd-43e8-b538-9ee89fdbf1b4", "id": 2}] 2024-09-18T14:21:20.502 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.495+0000 7fb13759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.495+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "07bd6e83-04fd-43e8-b538-9ee89fdbf1b4", "id": 2} v 0) v1 -- 0x7fb13812f340 con 0x7fb13809c9a0 2024-09-18T14:21:20.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb1267fc700 1 -- 172.21.15.174:0/359275278 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07bd6e83-04fd-43e8-b538-9ee89fdbf1b4", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb13001bbd0 con 0x7fb13809c9a0 2024-09-18T14:21:20.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb1267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "07bd6e83-04fd-43e8-b538-9ee89fdbf1b4", "id": 2}] 2024-09-18T14:21:20.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb1267fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:20.513 INFO:teuthology.orchestra.run.smithi174.stdout:2 2024-09-18T14:21:20.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 10 monclient: shutdown 2024-09-18T14:21:20.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 msgr2=0x7fb13812ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 0x7fb13812ebb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb13000ccc0 tx=0x7fb130005400 comp rx=0 tx=0).stop 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 shutdown_connections 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb13809c9a0 0x7fb13812ebb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13809bab0 0x7fb138118ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 --2- 172.21.15.174:0/359275278 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb13809b610 0x7fb1380676a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 >> 172.21.15.174:0/359275278 conn(0x7fb1380a1fb0 msgr2=0x7fb138131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 shutdown_connections 2024-09-18T14:21:20.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.507+0000 7fb13759e700 1 -- 172.21.15.174:0/359275278 wait complete. 2024-09-18T14:21:20.527 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 2876367d-d252-4764-b433-9cd2c82448b2 3 2024-09-18T14:21:20.617 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient: get_monmap_and_config 2024-09-18T14:21:20.617 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.617 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient: monmap: 2024-09-18T14:21:20.617 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.617 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:20.615084+0000 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:20.615084+0000 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:20.618 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 Processor -- start 2024-09-18T14:21:20.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 -- start start 2024-09-18T14:21:20.619 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient: init 2024-09-18T14:21:20.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:20.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T14:21:20.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): picked mon.noname-b con 0x7f6c840a2850 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:20.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c8409c240 0x7f6c84098a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.621 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): picked mon.noname-c con 0x7f6c8409c240 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6c840676a0 0x7f6c84067b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): picked mon.noname-a con 0x7f6c840676a0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c840a2cb0 con 0x7f6c840676a0 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c84098fc0 con 0x7f6c8409c240 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8409d9c0 con 0x7f6c840a2850 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8abf9700 10 monclient(hunting): authenticate will time out at 798.863439s 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c8409c240 0x7f6c84098a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c8409c240 0x7f6c84098a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43256/0 (socket says 172.21.15.174:43256) 2024-09-18T14:21:20.622 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 -- 172.21.15.174:0/3625176835 learned_addr learned my addr 172.21.15.174:0/3625176835 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): get_auth_request con 0x7f6c8409c240 auth_method 0 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8a3f8700 1 --2- 172.21.15.174:0/3625176835 >> v2:172.21.15.104:3300/0 conn(0x7f6c840676a0 0x7f6c84067b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c89bf7700 1 --2- 172.21.15.174:0/3625176835 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8a3f8700 10 monclient(hunting): get_auth_request con 0x7f6c840676a0 auth_method 0 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8a3f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8a3f8700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c8a3f8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c89bf7700 10 monclient(hunting): get_auth_request con 0x7f6c840a2850 auth_method 0 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c89bf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c89bf7700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c89bf7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 -- 172.21.15.174:0/3625176835 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 msgr2=0x7f6c8412fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 --2- 172.21.15.174:0/3625176835 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 -- 172.21.15.174:0/3625176835 >> v2:172.21.15.104:3300/0 conn(0x7f6c840676a0 msgr2=0x7f6c84067b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 --2- 172.21.15.174:0/3625176835 >> v2:172.21.15.104:3300/0 conn(0x7f6c840676a0 0x7f6c84067b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient: _finish_hunting 0 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 monclient: found mon.noname-c 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:20.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.611+0000 7f6c893f6700 1 -- 172.21.15.174:0/3625176835 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8409dca0 con 0x7f6c8409c240 2024-09-18T14:21:20.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c893f6700 1 -- 172.21.15.174:0/3625176835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 msgr2=0x7f6c84098a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c893f6700 1 --2- 172.21.15.174:0/3625176835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c84098a80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6c8409c6c0 tx=0x7f6c74009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b837a03bb72b5a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:20.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 1 -- 172.21.15.174:0/3625176835 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7400ba70 con 0x7f6c8409c240 2024-09-18T14:21:20.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: dump: 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.630 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: _finish_auth 0 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 5 monclient: authenticate success, global_id 4129 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 1 -- 172.21.15.174:0/3625176835 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7400bbd0 con 0x7f6c8409c240 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 4 set_mon_vals no callback set 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 1 -- 172.21.15.174:0/3625176835 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74011420 con 0x7f6c8409c240 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c88bf5700 10 monclient: dump: 2024-09-18T14:21:20.631 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 10 monclient: shutdown 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3625176835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 msgr2=0x7f6c84098a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3625176835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c84098a80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6c8409c6c0 tx=0x7f6c74009250 comp rx=0 tx=0).stop 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3625176835 shutdown_connections 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3625176835 >> v2:172.21.15.104:3300/0 conn(0x7f6c840676a0 0x7f6c84067b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3625176835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c84098a80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3625176835 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3625176835 >> 172.21.15.174:0/3625176835 conn(0x7f6c84068dc0 msgr2=0x7f6c840a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3625176835 shutdown_connections 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.615+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3625176835 wait complete. 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient: monmap: 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-09-18T14:21:20.633 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 Processor -- start 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 -- start start 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient: init 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c840676a0 0x7f6c8406dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): picked mon.noname-a con 0x7f6c840676a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:20.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c8411cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): picked mon.noname-b con 0x7f6c8409c240 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): picked mon.noname-c con 0x7f6c840a2850 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8411d500 con 0x7f6c840676a0 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8411df50 con 0x7f6c8409c240 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8411eeb0 con 0x7f6c840a2850 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8abf9700 10 monclient(hunting): authenticate will time out at 798.870665s 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c8411cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c8411cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43258/0 (socket says 172.21.15.174:43258) 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 -- 172.21.15.174:0/3096186805 learned_addr learned my addr 172.21.15.174:0/3096186805 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): get_auth_request con 0x7f6c8409c240 auth_method 0 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c840676a0 0x7f6c8406dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 10 monclient(hunting): get_auth_request con 0x7f6c840676a0 auth_method 0 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.636 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 10 monclient(hunting): get_auth_request con 0x7f6c840a2850 auth_method 0 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c89bf7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c8a3f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 -- 172.21.15.174:0/3096186805 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 msgr2=0x7f6c8412ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 -- 172.21.15.174:0/3096186805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c840676a0 msgr2=0x7f6c8406dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c840676a0 0x7f6c8406dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient: _finish_hunting 0 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 monclient: found mon.noname-b 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.619+0000 7f6c893f6700 1 -- 172.21.15.174:0/3096186805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740094f0 con 0x7f6c8409c240 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c893f6700 1 -- 172.21.15.174:0/3096186805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 msgr2=0x7f6c8411cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c893f6700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c8411cfc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c74006fd0 tx=0x7f6c74017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 1 -- 172.21.15.174:0/3096186805 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7401f070 con 0x7f6c8409c240 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: dump: 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.640 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 5 monclient: authenticate success, global_id 4132 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 1 -- 172.21.15.174:0/3096186805 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c74011e80 con 0x7f6c8409c240 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:20.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 1 -- 172.21.15.174:0/3096186805 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74006b10 con 0x7f6c8409c240 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 10 monclient: _renew_subs 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c7bfff700 4 set_mon_vals no callback set 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8411f190 con 0x7f6c8409c240 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 10 monclient: dump: 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.642 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 10 monclient: _renew_subs 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8411f750 con 0x7f6c8409c240 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 1 -- 172.21.15.174:0/3096186805 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f6c7401aad0 con 0x7f6c8409c240 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c79ffb700 1 -- 172.21.15.174:0/3096186805 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f6c7401ad80 con 0x7f6c8409c240 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.623+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c84063340 con 0x7f6c8409c240 2024-09-18T14:21:20.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.627+0000 7f6c79ffb700 1 -- 172.21.15.174:0/3096186805 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f6c74028070 con 0x7f6c8409c240 2024-09-18T14:21:20.644 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.627+0000 7f6c79ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:20.644 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.627+0000 7f6c79ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:20.712 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.703+0000 7f6c8abf9700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2876367d-d252-4764-b433-9cd2c82448b2", "id": 3}] 2024-09-18T14:21:20.712 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.703+0000 7f6c8abf9700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2876367d-d252-4764-b433-9cd2c82448b2", "id": 3}] 2024-09-18T14:21:20.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.703+0000 7f6c8abf9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:20.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.703+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2876367d-d252-4764-b433-9cd2c82448b2", "id": 3} v 0) v1 -- 0x7f6c8409ca80 con 0x7f6c8409c240 2024-09-18T14:21:20.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c79ffb700 1 -- 172.21.15.174:0/3096186805 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2876367d-d252-4764-b433-9cd2c82448b2", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6c74015020 con 0x7f6c8409c240 2024-09-18T14:21:20.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c79ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2876367d-d252-4764-b433-9cd2c82448b2", "id": 3}] 2024-09-18T14:21:20.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c79ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:20.866 INFO:teuthology.orchestra.run.smithi174.stdout:3 2024-09-18T14:21:20.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 10 monclient: shutdown 2024-09-18T14:21:20.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 msgr2=0x7f6c8411cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c8411cfc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c74006fd0 tx=0x7f6c74017000 comp rx=0 tx=0).stop 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 shutdown_connections 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6c840a2850 0x7f6c8412ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c8409c240 0x7f6c8411cfc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 --2- 172.21.15.174:0/3096186805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c840676a0 0x7f6c8406dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 >> 172.21.15.174:0/3096186805 conn(0x7f6c84068dc0 msgr2=0x7f6c840a1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 shutdown_connections 2024-09-18T14:21:20.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.859+0000 7f6c8abf9700 1 -- 172.21.15.174:0/3096186805 wait complete. 2024-09-18T14:21:20.883 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0 4 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient: get_monmap_and_config 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient: monmap: 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:20.980610+0000 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:20.980610+0000 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:20.983 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:20.984 INFO:teuthology.orchestra.run.smithi174.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:20.984 INFO:teuthology.orchestra.run.smithi174.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:20.984 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.985 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 1 Processor -- start 2024-09-18T14:21:20.985 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 1 -- start start 2024-09-18T14:21:20.985 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient: init 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fecd009a6b0 0x7fecd0131be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient(hunting): picked mon.noname-a con 0x7fecd009a6b0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecd009aa80 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecd0099e60 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd009b9c0 con 0x7fecd0099e60 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd00a1280 con 0x7fecd009aa80 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd009b6e0 con 0x7fecd009a6b0 2024-09-18T14:21:20.986 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecd491f700 10 monclient(hunting): authenticate will time out at 799.227340s 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecceffd700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fecd009aa80 0x7fecd0132120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59190/0 (socket says 172.21.15.174:59190) 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7fecceffd700 1 -- 172.21.15.174:0/3659440676 learned_addr learned my addr 172.21.15.174:0/3659440676 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 1 --2- 172.21.15.174:0/3659440676 >> v2:172.21.15.104:3300/0 conn(0x7fecd009a6b0 0x7fecd0131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 10 monclient(hunting): get_auth_request con 0x7fecd009a6b0 auth_method 0 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:20.987 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.975+0000 7feccf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:20.988 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7fecceffd700 1 -- 172.21.15.174:0/3659440676 >> v1:172.21.15.174:6789/0 conn(0x7fecd009aa80 legacy=0x7fecd0132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.988 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7fecce7fc700 1 -- 172.21.15.174:0/3659440676 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851366240 0 0) 0x7fecd00a1280 con 0x7fecd009aa80 2024-09-18T14:21:20.988 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7fecce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7fecce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7fecce7fc700 10 monclient(hunting): my global_id is 4135 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccffff700 1 -- 172.21.15.174:0/3659440676 >> v1:172.21.15.104:6789/0 conn(0x7fecd0099e60 legacy=0x7fecd0138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7fecce7fc700 1 -- 172.21.15.174:0/3659440676 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb8003570 con 0x7fecd009aa80 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 10 monclient(hunting): handle_auth_done global_id 4134 payload 274 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 1 -- 172.21.15.174:0/3659440676 >> v1:172.21.15.174:6789/0 conn(0x7fecd009aa80 legacy=0x7fecd0132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 1 -- 172.21.15.174:0/3659440676 >> v1:172.21.15.104:6789/0 conn(0x7fecd0099e60 legacy=0x7fecd0138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 1 monclient: found mon.noname-a 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.979+0000 7feccf7fe700 1 -- 172.21.15.174:0/3659440676 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd0132830 con 0x7fecd009a6b0 2024-09-18T14:21:20.991 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 1 -- 172.21.15.174:0/3659440676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd009a6b0 msgr2=0x7fecd0131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 1 --2- 172.21.15.174:0/3659440676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd009a6b0 0x7fecd0131be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009f60 tx=0x7fecc8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3679b314060aa8f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 1 -- 172.21.15.174:0/3659440676 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc800e070 con 0x7fecd009a6b0 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: dump: 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.992 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 5 monclient: authenticate success, global_id 4134 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 1 -- 172.21.15.174:0/3659440676 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc800be40 con 0x7fecd009a6b0 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 1 -- 172.21.15.174:0/3659440676 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8012660 con 0x7fecd009a6b0 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 4 set_mon_vals no callback set 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:20.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecce7fc700 10 monclient: dump: 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient: shutdown 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- 172.21.15.174:0/3659440676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd009a6b0 msgr2=0x7fecd0131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:20.994 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 --2- 172.21.15.174:0/3659440676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd009a6b0 0x7fecd0131be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009f60 tx=0x7fecc8009780 comp rx=0 tx=0).stop 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- 172.21.15.174:0/3659440676 shutdown_connections 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 --2- 172.21.15.174:0/3659440676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd009a6b0 0x7fecd0131be0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- 172.21.15.174:0/3659440676 >> 172.21.15.174:0/3659440676 conn(0x7fecd00a53b0 msgr2=0x7fecd00a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- 172.21.15.174:0/3659440676 shutdown_connections 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- 172.21.15.174:0/3659440676 wait complete. 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient: build_initial_monmap 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient: monmap: 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:20.995 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:20.996 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:20.996 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:20.996 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:20.996 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 Processor -- start 2024-09-18T14:21:20.996 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- start start 2024-09-18T14:21:20.996 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient: init 2024-09-18T14:21:21.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:21.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:21.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 0x7fecd0131360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:21.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): picked mon.noname-b con 0x7fecd0099e60 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:21.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fecd009a6b0 0x7fecd0130870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:21.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): picked mon.noname-c con 0x7fecd009a6b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:21.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd006f570 0x7fecd0140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): picked mon.noname-a con 0x7fecd006f570 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd006cdc0 con 0x7fecd006f570 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd006d030 con 0x7fecd0099e60 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd009ae60 con 0x7fecd009a6b0 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecd491f700 10 monclient(hunting): authenticate will time out at 799.235565s 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 0x7fecd0131360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 0x7fecd0131360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43272/0 (socket says 172.21.15.174:43272) 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 1 -- 172.21.15.174:0/43312413 learned_addr learned my addr 172.21.15.174:0/43312413 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-18T14:21:21.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 10 monclient(hunting): get_auth_request con 0x7fecd0099e60 auth_method 0 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7feccffff700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd006f570 0x7fecd0140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.983+0000 7fecceffd700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fecd009a6b0 0x7fecd0130870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccffff700 10 monclient(hunting): get_auth_request con 0x7fecd006f570 auth_method 0 2024-09-18T14:21:21.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecceffd700 10 monclient(hunting): get_auth_request con 0x7fecd009a6b0 auth_method 0 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:21.003 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 -- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fecd009a6b0 msgr2=0x7fecd0130870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fecd009a6b0 0x7fecd0130870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 -- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd006f570 msgr2=0x7fecd0140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd006f570 0x7fecd0140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 monclient: found mon.noname-b 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 -- 172.21.15.174:0/43312413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8009a20 con 0x7fecd0099e60 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecceffd700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fecd009a6b0 0x7fecd0130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 -- 172.21.15.174:0/43312413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 msgr2=0x7fecd0131360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 0x7fecd0131360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009460 tx=0x7fecc8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:21.004 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 1 -- 172.21.15.174:0/43312413 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8012e30 con 0x7fecd0099e60 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: dump: 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:21.005 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 5 monclient: authenticate success, global_id 4138 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 10 monclient: _renew_subs 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 1 -- 172.21.15.174:0/43312413 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc801b630 con 0x7fecd0099e60 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd006b800 con 0x7fecd0099e60 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccf7fe700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 0x7fecd0131360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009460 tx=0x7fecc8017000 comp rx=0 tx=0).queue_received will delay after 1.72667e+09 on 0x7fecc801b9f0 mon_map magic: 0 v1 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7feccdffb700 4 set_mon_vals no callback set 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 10 monclient: _renew_subs 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd0130db0 con 0x7fecd0099e60 2024-09-18T14:21:21.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 1 -- 172.21.15.174:0/43312413 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc801b9f0 con 0x7fecd0099e60 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 10 monclient: dump: 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:21.007 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecb77fe700 1 -- 172.21.15.174:0/43312413 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fecc802f440 con 0x7fecd0099e60 2024-09-18T14:21:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.987+0000 7fecd491f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.991+0000 7fecd491f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.991+0000 7fecd491f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.991+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd0063340 con 0x7fecd0099e60 2024-09-18T14:21:21.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:20.991+0000 7fecb77fe700 1 -- 172.21.15.174:0/43312413 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fecc801bd60 con 0x7fecd0099e60 2024-09-18T14:21:21.124 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.115+0000 7ff37b32f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T14:21:21.181 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.175+0000 7ff37b32f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T14:21:21.295 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.287+0000 7ff37b32f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T14:21:21.534 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.527+0000 7ff37b32f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T14:21:21.584 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.575+0000 7ff37b32f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T14:21:21.770 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.763+0000 7ff37b32f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T14:21:21.824 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.815+0000 7fecb77fe700 1 -- 172.21.15.174:0/43312413 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7fecc8034060 con 0x7fecd0099e60 2024-09-18T14:21:21.824 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.815+0000 7fecb77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:21.824 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.815+0000 7fecb77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:21.833 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.827+0000 7ff37b32f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T14:21:21.894 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:21.887+0000 7ff37b32f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T14:21:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.895+0000 7fecd491f700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0", "id": 4}] 2024-09-18T14:21:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.895+0000 7fecd491f700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0", "id": 4}] 2024-09-18T14:21:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.895+0000 7fecd491f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.895+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0", "id": 4} v 0) v1 -- 0x7fecd00689e0 con 0x7fecd0099e60 2024-09-18T14:21:21.914 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecb77fe700 1 -- 172.21.15.174:0/43312413 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fecc802a070 con 0x7fecd0099e60 2024-09-18T14:21:21.914 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecb77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0", "id": 4}] 2024-09-18T14:21:21.914 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecb77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:21.914 INFO:teuthology.orchestra.run.smithi174.stdout:4 2024-09-18T14:21:21.915 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 10 monclient: shutdown 2024-09-18T14:21:21.915 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:21.915 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 msgr2=0x7fecd0131360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 0x7fecd0131360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009460 tx=0x7fecc8017000 comp rx=0 tx=0).stop 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 shutdown_connections 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecd006f570 0x7fecd0140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fecd009a6b0 0x7fecd0130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 --2- 172.21.15.174:0/43312413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecd0099e60 0x7fecd0131360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 >> 172.21.15.174:0/43312413 conn(0x7fecd00a53b0 msgr2=0x7fecd012beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 shutdown_connections 2024-09-18T14:21:21.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:21.907+0000 7fecd491f700 1 -- 172.21.15.174:0/43312413 wait complete. 2024-09-18T14:21:21.927 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 57b7828f-2f7a-431b-8ed4-726d1011dd97 5 2024-09-18T14:21:22.023 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.015+0000 7fbf035ef700 10 monclient: get_monmap_and_config 2024-09-18T14:21:22.023 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.015+0000 7fbf035ef700 10 monclient: build_initial_monmap 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.015+0000 7fbf035ef700 10 monclient: monmap: 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:22.021174+0000 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:22.021174+0000 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:22.024 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.015+0000 7fbf035ef700 1 Processor -- start 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.015+0000 7fbf035ef700 1 -- start start 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.015+0000 7fbf035ef700 10 monclient: init 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbefc0676a0 0x7fbefc09e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient(hunting): picked mon.noname-c con 0x7fbefc0676a0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbefc06f1d0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbefc09d410 0x7fbefc139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient(hunting): picked mon.noname-a con 0x7fbefc09d410 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc09ec20 con 0x7fbefc06f1d0 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbefc099cb0 con 0x7fbefc09d410 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbefc067a70 con 0x7fbefc0676a0 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient(hunting): authenticate will time out at 800.268496s 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbefc0676a0 0x7fbefc09e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf00b8a700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fbefc06f1d0 0x7fbefc09cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59206/0 (socket says 172.21.15.174:59206) 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf00b8a700 1 -- 172.21.15.174:0/2134837765 learned_addr learned my addr 172.21.15.174:0/2134837765 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbefc0676a0 0x7fbefc09e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43278/0 (socket says 172.21.15.174:43278) 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): get_auth_request con 0x7fbefc0676a0 auth_method 0 2024-09-18T14:21:22.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf01b8c700 1 --2- 172.21.15.174:0/2134837765 >> v2:172.21.15.104:3300/0 conn(0x7fbefc09d410 0x7fbefc139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 -- 172.21.15.174:0/2134837765 >> v2:172.21.15.104:3300/0 conn(0x7fbefc09d410 msgr2=0x7fbefc139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 --2- 172.21.15.174:0/2134837765 >> v2:172.21.15.104:3300/0 conn(0x7fbefc09d410 0x7fbefc139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 -- 172.21.15.174:0/2134837765 >> v1:172.21.15.174:6789/0 conn(0x7fbefc06f1d0 legacy=0x7fbefc09cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient: _finish_hunting 0 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 monclient: found mon.noname-c 2024-09-18T14:21:22.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:22.030 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:22.030 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 -- 172.21.15.174:0/2134837765 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc0999d0 con 0x7fbefc0676a0 2024-09-18T14:21:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 -- 172.21.15.174:0/2134837765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc0676a0 msgr2=0x7fbefc09e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf0138b700 1 --2- 172.21.15.174:0/2134837765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc0676a0 0x7fbefc09e6e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbefc068580 tx=0x7fbef80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23822f6061e07978 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 1 -- 172.21.15.174:0/2134837765 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef800b9a0 con 0x7fbefc0676a0 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: dump: 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.036 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 5 monclient: authenticate success, global_id 4141 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 1 -- 172.21.15.174:0/2134837765 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef800bb00 con 0x7fbefc0676a0 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 1 -- 172.21.15.174:0/2134837765 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef800bea0 con 0x7fbefc0676a0 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbf035ef700 4 set_mon_vals no callback set 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.019+0000 7fbef3fff700 10 monclient: dump: 2024-09-18T14:21:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient: shutdown 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- 172.21.15.174:0/2134837765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc0676a0 msgr2=0x7fbefc09e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 --2- 172.21.15.174:0/2134837765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc0676a0 0x7fbefc09e6e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbefc068580 tx=0x7fbef80091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.038 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- 172.21.15.174:0/2134837765 shutdown_connections 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 --2- 172.21.15.174:0/2134837765 >> v2:172.21.15.104:3300/0 conn(0x7fbefc09d410 0x7fbefc139f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 --2- 172.21.15.174:0/2134837765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc0676a0 0x7fbefc09e6e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- 172.21.15.174:0/2134837765 >> 172.21.15.174:0/2134837765 conn(0x7fbefc069270 msgr2=0x7fbefc0a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- 172.21.15.174:0/2134837765 shutdown_connections 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- 172.21.15.174:0/2134837765 wait complete. 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient: build_initial_monmap 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient: monmap: 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:22.039 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 Processor -- start 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- start start 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient: init 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbefc0676a0 0x7fbefc141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): picked mon.noname-a con 0x7fbefc0676a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 0x7fbefc134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): picked mon.noname-b con 0x7fbefc06f1d0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbefc09d410 0x7fbefc11daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): picked mon.noname-c con 0x7fbefc09d410 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc121470 con 0x7fbefc0676a0 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc09c080 con 0x7fbefc06f1d0 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc134fe0 con 0x7fbefc09d410 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf035ef700 10 monclient(hunting): authenticate will time out at 800.274721s 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 0x7fbefc134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 0x7fbefc134070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43280/0 (socket says 172.21.15.174:43280) 2024-09-18T14:21:22.041 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 -- 172.21.15.174:0/4225630147 learned_addr learned my addr 172.21.15.174:0/4225630147 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): get_auth_request con 0x7fbefc06f1d0 auth_method 0 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbefc09d410 0x7fbefc11daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbefc0676a0 0x7fbefc141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 10 monclient(hunting): get_auth_request con 0x7fbefc09d410 auth_method 0 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 10 monclient(hunting): get_auth_request con 0x7fbefc0676a0 auth_method 0 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf01b8c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf0138b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 -- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbefc09d410 msgr2=0x7fbefc11daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbefc09d410 0x7fbefc11daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 -- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbefc0676a0 msgr2=0x7fbefc141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbefc0676a0 0x7fbefc141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient: _finish_hunting 0 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 monclient: found mon.noname-b 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.023+0000 7fbf00b8a700 1 -- 172.21.15.174:0/4225630147 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8009420 con 0x7fbefc06f1d0 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf00b8a700 1 -- 172.21.15.174:0/4225630147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 msgr2=0x7fbefc134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf00b8a700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 0x7fbefc134070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c5e0 tx=0x7fbeec004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 1 -- 172.21.15.174:0/4225630147 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec005d40 con 0x7fbefc06f1d0 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:22.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: dump: 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: _finish_auth 0 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 5 monclient: authenticate success, global_id 4144 2024-09-18T14:21:22.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 1 -- 172.21.15.174:0/4225630147 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeec0070d0 con 0x7fbefc06f1d0 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 10 monclient: _renew_subs 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbefc1423e0 con 0x7fbefc06f1d0 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 1 -- 172.21.15.174:0/4225630147 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec007490 con 0x7fbefc06f1d0 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 10 monclient: dump: 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.046 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef37fe700 4 set_mon_vals no callback set 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 10 monclient: _renew_subs 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbefc11f6d0 con 0x7fbefc06f1d0 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 1 -- 172.21.15.174:0/4225630147 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fbeec01c440 con 0x7fbefc06f1d0 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbef17fa700 1 -- 172.21.15.174:0/4225630147 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fbeec012980 con 0x7fbefc06f1d0 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:22.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.027+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbefc0636d0 con 0x7fbefc06f1d0 2024-09-18T14:21:22.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.031+0000 7fbef17fa700 1 -- 172.21.15.174:0/4225630147 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7fbeec017070 con 0x7fbefc06f1d0 2024-09-18T14:21:22.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.031+0000 7fbef17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:22.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.031+0000 7fbef17fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:22.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.103+0000 7fbf035ef700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "57b7828f-2f7a-431b-8ed4-726d1011dd97", "id": 5}] 2024-09-18T14:21:22.113 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.103+0000 7fbf035ef700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "57b7828f-2f7a-431b-8ed4-726d1011dd97", "id": 5}] 2024-09-18T14:21:22.113 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.103+0000 7fbf035ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:22.113 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.103+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "57b7828f-2f7a-431b-8ed4-726d1011dd97", "id": 5} v 0) v1 -- 0x7fbefc0a3170 con 0x7fbefc06f1d0 2024-09-18T14:21:22.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbef17fa700 1 -- 172.21.15.174:0/4225630147 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "57b7828f-2f7a-431b-8ed4-726d1011dd97", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbeec01c920 con 0x7fbefc06f1d0 2024-09-18T14:21:22.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbef17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "57b7828f-2f7a-431b-8ed4-726d1011dd97", "id": 5}] 2024-09-18T14:21:22.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbef17fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:22.124 INFO:teuthology.orchestra.run.smithi174.stdout:5 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 10 monclient: shutdown 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 msgr2=0x7fbefc134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 0x7fbefc134070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c5e0 tx=0x7fbeec004a80 comp rx=0 tx=0).stop 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 shutdown_connections 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbefc09d410 0x7fbefc11daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbefc06f1d0 0x7fbefc134070 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 --2- 172.21.15.174:0/4225630147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbefc0676a0 0x7fbefc141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 >> 172.21.15.174:0/4225630147 conn(0x7fbefc069270 msgr2=0x7fbefc097b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 shutdown_connections 2024-09-18T14:21:22.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-18T14:21:22.115+0000 7fbf035ef700 1 -- 172.21.15.174:0/4225630147 wait complete. 2024-09-18T14:21:22.137 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T14:21:22.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T14:21:22.139 INFO:tasks.ceph.osd.0:Started 2024-09-18T14:21:22.139 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T14:21:22.140 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T14:21:22.142 INFO:tasks.ceph.osd.1:Started 2024-09-18T14:21:22.142 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T14:21:22.142 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T14:21:22.145 INFO:tasks.ceph.osd.2:Started 2024-09-18T14:21:22.145 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T14:21:22.145 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T14:21:22.147 INFO:tasks.ceph.osd.3:Started 2024-09-18T14:21:22.147 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-18T14:21:22.147 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-18T14:21:22.149 INFO:tasks.ceph.osd.4:Started 2024-09-18T14:21:22.149 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-18T14:21:22.149 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-18T14:21:22.151 INFO:tasks.ceph.osd.5:Started 2024-09-18T14:21:22.151 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:21:22.211 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-18T14:21:22.203+0000 7ff2b5277540 -1 Falling back to public interface 2024-09-18T14:21:22.213 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-18T14:21:22.207+0000 7f2b62825540 -1 Falling back to public interface 2024-09-18T14:21:22.215 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-09-18T14:21:22.208+0000 7fa8cad59540 -1 Falling back to public interface 2024-09-18T14:21:22.215 INFO:tasks.ceph.osd.3.smithi174.stderr:2024-09-18T14:21:22.207+0000 7f070b7ca540 -1 Falling back to public interface 2024-09-18T14:21:22.219 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-09-18T14:21:22.212+0000 7f49a6333540 -1 Falling back to public interface 2024-09-18T14:21:22.223 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-18T14:21:22.215+0000 7f2b62825540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:22.223 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-18T14:21:22.215+0000 7ff2b5277540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:22.224 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-18T14:21:22.215+0000 7f2b62825540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.224 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-18T14:21:22.215+0000 7f2b62825540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.224 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-18T14:21:22.215+0000 7ff2b5277540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.224 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-18T14:21:22.215+0000 7ff2b5277540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.230 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-09-18T14:21:22.220+0000 7fa8cad59540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:22.230 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-09-18T14:21:22.220+0000 7fa8cad59540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.230 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-09-18T14:21:22.220+0000 7fa8cad59540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.231 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-18T14:21:22.223+0000 7f2b62825540 -1 osd.5 0 log_to_monitors true 2024-09-18T14:21:22.231 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-18T14:21:22.223+0000 7ff2b5277540 -1 osd.4 0 log_to_monitors true 2024-09-18T14:21:22.232 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-09-18T14:21:22.224+0000 7f1a3dfa7540 -1 Falling back to public interface 2024-09-18T14:21:22.235 INFO:tasks.ceph.osd.3.smithi174.stderr:2024-09-18T14:21:22.227+0000 7f070b7ca540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:22.235 INFO:tasks.ceph.osd.3.smithi174.stderr:2024-09-18T14:21:22.227+0000 7f070b7ca540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.235 INFO:tasks.ceph.osd.3.smithi174.stderr:2024-09-18T14:21:22.227+0000 7f070b7ca540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.236 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-09-18T14:21:22.228+0000 7fa8cad59540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:21:22.237 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-09-18T14:21:22.228+0000 7f49a6333540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:22.237 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-09-18T14:21:22.228+0000 7f49a6333540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.237 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-09-18T14:21:22.228+0000 7f49a6333540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.239 INFO:tasks.ceph.osd.3.smithi174.stderr:2024-09-18T14:21:22.231+0000 7f070b7ca540 -1 osd.3 0 log_to_monitors true 2024-09-18T14:21:22.244 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-09-18T14:21:22.236+0000 7f49a6333540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:21:22.249 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-09-18T14:21:22.240+0000 7f1a3dfa7540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:21:22.249 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-09-18T14:21:22.240+0000 7f1a3dfa7540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.249 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-09-18T14:21:22.240+0000 7f1a3dfa7540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:21:22.254 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-09-18T14:21:22.248+0000 7f1a3dfa7540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.256+0000 7ff31ccba700 10 monclient: get_monmap_and_config 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.256+0000 7ff31ccba700 10 monclient: build_initial_monmap 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.256+0000 7ff31ccba700 10 monclient: monmap: 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:22.263488+0000 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:22.263488+0000 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:22.267 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 Processor -- start 2024-09-18T14:21:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 -- start start 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient: init 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 0x7ff318099340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): picked mon.noname-b con 0x7ff31809d7c0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff31809ac30 0x7ff31809b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): picked mon.noname-c con 0x7ff31809ac30 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff318095a50 0x7ff318095e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): picked mon.noname-a con 0x7ff318095a50 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31809bfb0 con 0x7ff318095a50 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff318098110 con 0x7ff31809ac30 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff318097e30 con 0x7ff31809d7c0 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31ccba700 10 monclient(hunting): authenticate will time out at 803.253402s 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 0x7ff318099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 0x7ff318099340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:38738/0 (socket says 172.21.15.104:38738) 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 -- 172.21.15.104:0/343395855 learned_addr learned my addr 172.21.15.104:0/343395855 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): get_auth_request con 0x7ff31809d7c0 auth_method 0 2024-09-18T14:21:22.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 1 --2- 172.21.15.104:0/343395855 >> v2:172.21.15.104:3300/0 conn(0x7ff318095a50 0x7ff318095e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 10 monclient(hunting): get_auth_request con 0x7ff318095a50 auth_method 0 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff315d9b700 1 --2- 172.21.15.104:0/343395855 >> v2:172.21.15.174:3300/0 conn(0x7ff31809ac30 0x7ff31809b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff315d9b700 10 monclient(hunting): get_auth_request con 0x7ff31809ac30 auth_method 0 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff315d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff315d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff315d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.273 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient(hunting): handle_auth_done global_id 4166 payload 274 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 -- 172.21.15.104:0/343395855 >> v2:172.21.15.174:3300/0 conn(0x7ff31809ac30 msgr2=0x7ff31809b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 --2- 172.21.15.104:0/343395855 >> v2:172.21.15.174:3300/0 conn(0x7ff31809ac30 0x7ff31809b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 -- 172.21.15.104:0/343395855 >> v2:172.21.15.104:3300/0 conn(0x7ff318095a50 msgr2=0x7ff318095e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 --2- 172.21.15.104:0/343395855 >> v2:172.21.15.104:3300/0 conn(0x7ff318095a50 0x7ff318095e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 monclient: found mon.noname-b 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff31659c700 1 -- 172.21.15.104:0/343395855 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff318097b50 con 0x7ff31809d7c0 2024-09-18T14:21:22.274 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.260+0000 7ff316d9d700 1 --2- 172.21.15.104:0/343395855 >> v2:172.21.15.104:3300/0 conn(0x7ff318095a50 0x7ff318095e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:22.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 1 -- 172.21.15.104:0/343395855 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 msgr2=0x7ff318099340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 1 --2- 172.21.15.104:0/343395855 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 0x7ff318099340 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff31806a7d0 tx=0x7ff30800d6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=764037cb8b1f04dd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 1 -- 172.21.15.104:0/343395855 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308012070 con 0x7ff31809d7c0 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: dump: 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:22.280 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: _finish_auth 0 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 5 monclient: authenticate success, global_id 4166 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 1 -- 172.21.15.104:0/343395855 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30800d2e0 con 0x7ff31809d7c0 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:22.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 4 set_mon_vals no callback set 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 1 -- 172.21.15.104:0/343395855 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30800fd50 con 0x7ff31809d7c0 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31559a700 10 monclient: dump: 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.282 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient: shutdown 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- 172.21.15.104:0/343395855 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 msgr2=0x7ff318099340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 --2- 172.21.15.104:0/343395855 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 0x7ff318099340 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff31806a7d0 tx=0x7ff30800d6f0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- 172.21.15.104:0/343395855 shutdown_connections 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 --2- 172.21.15.104:0/343395855 >> v2:172.21.15.104:3300/0 conn(0x7ff318095a50 0x7ff318095e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 --2- 172.21.15.104:0/343395855 >> v2:172.21.15.174:3300/0 conn(0x7ff31809ac30 0x7ff31809b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 --2- 172.21.15.104:0/343395855 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809d7c0 0x7ff318099340 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- 172.21.15.104:0/343395855 >> 172.21.15.104:0/343395855 conn(0x7ff3180a1290 msgr2=0x7ff3180a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- 172.21.15.104:0/343395855 shutdown_connections 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- 172.21.15.104:0/343395855 wait complete. 2024-09-18T14:21:22.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient: build_initial_monmap 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient: monmap: 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 Processor -- start 2024-09-18T14:21:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- start start 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient: init 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 0x7ff318139680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): picked mon.noname-a con 0x7ff318095a50 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809ac30 0x7ff318138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): picked mon.noname-c con 0x7ff31809ac30 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31809d7c0 0x7ff31813cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): picked mon.noname-b con 0x7ff31809d7c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31811e670 con 0x7ff318095a50 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31811e0b0 con 0x7ff31809d7c0 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31811bb40 con 0x7ff31809ac30 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31ccba700 10 monclient(hunting): authenticate will time out at 803.259553s 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 0x7ff318139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809ac30 0x7ff318138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 0x7ff318139680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52000/0 (socket says 172.21.15.104:52000) 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 1 -- 172.21.15.104:0/2413947673 learned_addr learned my addr 172.21.15.104:0/2413947673 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 10 monclient(hunting): get_auth_request con 0x7ff318095a50 auth_method 0 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 10 monclient(hunting): get_auth_request con 0x7ff31809ac30 auth_method 0 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31809d7c0 0x7ff31813cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff31659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 10 monclient(hunting): get_auth_request con 0x7ff31809d7c0 auth_method 0 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff315d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.264+0000 7ff316d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 10 monclient(hunting): handle_auth_done global_id 4176 payload 824 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 -- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809ac30 msgr2=0x7ff318138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809ac30 0x7ff318138d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 -- 172.21.15.104:0/2413947673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31809d7c0 msgr2=0x7ff31813cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31809d7c0 0x7ff31813cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 monclient: found mon.noname-a 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:22.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 -- 172.21.15.104:0/2413947673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c009420 con 0x7ff318095a50 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 -- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 msgr2=0x7ff318139680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31659c700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 0x7ff318139680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff308006fd0 tx=0x7ff30801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 1 -- 172.21.15.104:0/2413947673 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30800fd50 con 0x7ff318095a50 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: dump: 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.289 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31ccba700 5 monclient: authenticate success, global_id 4176 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31ccba700 10 monclient: _renew_subs 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 1 -- 172.21.15.104:0/2413947673 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff308025070 con 0x7ff318095a50 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31ccba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30800d990 con 0x7ff318095a50 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 1 -- 172.21.15.104:0/2413947673 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30800b890 con 0x7ff318095a50 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff314d99700 4 set_mon_vals no callback set 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 10 monclient: dump: 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31ccba700 10 monclient: _renew_subs 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31ccba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:22.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff318139270 con 0x7ff318095a50 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 1 -- 172.21.15.104:0/2413947673 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ff308027030 con 0x7ff318095a50 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.268+0000 7ff3067fc700 1 -- 172.21.15.104:0/2413947673 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7ff308021420 con 0x7ff318095a50 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.272+0000 7ff31ccba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.272+0000 7ff31ccba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.272+0000 7ff31ccba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.272+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31811dee0 con 0x7ff318095a50 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.276+0000 7ff3067fc700 1 -- 172.21.15.104:0/2413947673 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7ff30802a430 con 0x7ff318095a50 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.276+0000 7ff3067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:22.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.276+0000 7ff3067fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:22.302 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:22.295+0000 7ff37b32f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T14:21:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.340+0000 7ff31ccba700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.340+0000 7ff31ccba700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.340+0000 7ff31ccba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.340+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff318064b30 con 0x7ff318095a50 2024-09-18T14:21:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.340+0000 7ff3067fc700 1 -- 172.21.15.104:0/2413947673 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7ff308021630 con 0x7ff318095a50 2024-09-18T14:21:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.340+0000 7ff3067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.340+0000 7ff3067fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:22.350 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:22.350 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":8,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:21:22.111706+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 10 monclient: shutdown 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 msgr2=0x7ff318139680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 0x7ff318139680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff308006fd0 tx=0x7ff30801c000 comp rx=0 tx=0).stop 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 shutdown_connections 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff31809d7c0 0x7ff31813cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff31809ac30 0x7ff318138d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 --2- 172.21.15.104:0/2413947673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff318095a50 0x7ff318139680 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 >> 172.21.15.104:0/2413947673 conn(0x7ff3180a1290 msgr2=0x7ff31809fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 shutdown_connections 2024-09-18T14:21:22.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:22.344+0000 7ff31ccba700 1 -- 172.21.15.104:0/2413947673 wait complete. 2024-09-18T14:21:22.362 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T14:21:22.362 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T14:21:22.415 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:22.407+0000 7ff37b32f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T14:21:22.708 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:22.699+0000 7ff37b32f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T14:21:22.873 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:22.863+0000 7ff37b32f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T14:21:22.941 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:22.935+0000 7ff37b32f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T14:21:22.967 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T14:21:22.993 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:22.987+0000 7ff37b32f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T14:21:23.051 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:23.043+0000 7ff37b32f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.060+0000 7f0ec8a5c700 10 monclient: get_monmap_and_config 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.060+0000 7f0ec8a5c700 10 monclient: build_initial_monmap 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.060+0000 7f0ec8a5c700 10 monclient: monmap: 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:23.067550+0000 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:23.067550+0000 2024-09-18T14:21:23.070 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:23.071 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:23.071 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:23.071 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:23.071 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:23.071 INFO:teuthology.misc.health.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:23.071 INFO:teuthology.misc.health.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:23.071 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:23.073 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 Processor -- start 2024-09-18T14:21:23.073 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 -- start start 2024-09-18T14:21:23.073 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient: init 2024-09-18T14:21:23.073 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:23.073 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T14:21:23.073 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0ec40a2030 0x7f0ec409ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:23.073 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): picked mon.noname-a con 0x7f0ec40a2030 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec409f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): picked mon.noname-b con 0x7f0ec409ecc0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0ec4131af0 0x7f0ec4131f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): picked mon.noname-c con 0x7f0ec4131af0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec409f5d0 con 0x7f0ec40a2030 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec409d790 con 0x7f0ec4131af0 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec409cf60 con 0x7f0ec409ecc0 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec8a5c700 10 monclient(hunting): authenticate will time out at 804.057308s 2024-09-18T14:21:23.074 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec409f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0ec40a2030 0x7f0ec409ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec409f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:38750/0 (socket says 172.21.15.104:38750) 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 1 -- 172.21.15.104:0/449662539 learned_addr learned my addr 172.21.15.104:0/449662539 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 10 monclient(hunting): get_auth_request con 0x7f0ec409ecc0 auth_method 0 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): get_auth_request con 0x7f0ec40a2030 auth_method 0 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:23.075 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 1 --2- 172.21.15.104:0/449662539 >> v2:172.21.15.174:3300/0 conn(0x7f0ec4131af0 0x7f0ec4131f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 10 monclient(hunting): get_auth_request con 0x7f0ec4131af0 auth_method 0 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:23.076 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient(hunting): handle_auth_done global_id 4179 payload 274 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 1 -- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 msgr2=0x7f0ec409f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 1 --2- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec409f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 1 -- 172.21.15.104:0/449662539 >> v2:172.21.15.174:3300/0 conn(0x7f0ec4131af0 msgr2=0x7f0ec4131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 1 --2- 172.21.15.104:0/449662539 >> v2:172.21.15.174:3300/0 conn(0x7f0ec4131af0 0x7f0ec4131f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 1 monclient: found mon.noname-a 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec2d9d700 1 -- 172.21.15.104:0/449662539 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec409d1d0 con 0x7f0ec40a2030 2024-09-18T14:21:23.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.064+0000 7f0ec259c700 1 --2- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec409f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:23.080 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 1 -- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec40a2030 msgr2=0x7f0ec409ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:23.080 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 1 --2- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec40a2030 0x7f0ec409ff00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40a1460 tx=0x7f0eac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d651603197e4272 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:23.080 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 1 -- 172.21.15.104:0/449662539 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac011840 con 0x7f0ec40a2030 2024-09-18T14:21:23.080 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:23.080 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: dump: 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: _finish_auth 0 2024-09-18T14:21:23.081 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 5 monclient: authenticate success, global_id 4179 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 1 -- 172.21.15.104:0/449662539 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac0119c0 con 0x7f0ec40a2030 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 4 set_mon_vals no callback set 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 1 -- 172.21.15.104:0/449662539 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac011d80 con 0x7f0ec40a2030 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec1d9b700 10 monclient: dump: 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.082 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient: shutdown 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec40a2030 msgr2=0x7f0ec409ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec40a2030 0x7f0ec409ff00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40a1460 tx=0x7f0eac0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/449662539 shutdown_connections 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/449662539 >> v2:172.21.15.174:3300/0 conn(0x7f0ec4131af0 0x7f0ec4131f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec409f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/449662539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec40a2030 0x7f0ec409ff00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/449662539 >> 172.21.15.104:0/449662539 conn(0x7f0ec40635a0 msgr2=0x7f0ec40639a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/449662539 shutdown_connections 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/449662539 wait complete. 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient: build_initial_monmap 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient: monmap: 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:23.084 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 Processor -- start 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- start start 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient: init 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec41413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): picked mon.noname-c con 0x7f0ec409ecc0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ec40a2030 0x7f0ec412f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): picked mon.noname-b con 0x7f0ec40a2030 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:23.085 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec4131af0 0x7f0ec411cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): picked mon.noname-a con 0x7f0ec4131af0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec409bf00 con 0x7f0ec4131af0 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec4131680 con 0x7f0ec40a2030 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec411f270 con 0x7f0ec409ecc0 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec8a5c700 10 monclient(hunting): authenticate will time out at 804.063233s 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec41413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec41413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:38760/0 (socket says 172.21.15.104:38760) 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 1 -- 172.21.15.104:0/2419139817 learned_addr learned my addr 172.21.15.104:0/2419139817 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:23.086 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 10 monclient(hunting): get_auth_request con 0x7f0ec409ecc0 auth_method 0 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec4131af0 0x7f0ec411cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): get_auth_request con 0x7f0ec4131af0 auth_method 0 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:23.087 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient(hunting): handle_auth_done global_id 4182 payload 824 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec259c700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ec40a2030 0x7f0ec412f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 1 -- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 msgr2=0x7f0ec41413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec41413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 1 -- 172.21.15.104:0/2419139817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ec40a2030 msgr2=0x7f0ec412f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ec40a2030 0x7f0ec412f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient: _finish_hunting 0 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 1 monclient: found mon.noname-a 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec359e700 1 -- 172.21.15.104:0/2419139817 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb40094f0 con 0x7f0ec4131af0 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec259c700 10 monclient: get_auth_request con 0x7f0ec40a2030 auth_method 0 2024-09-18T14:21:23.088 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.068+0000 7f0ec259c700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ec40a2030 0x7f0ec412f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:21:23.089 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.072+0000 7f0ec359e700 1 -- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec4131af0 msgr2=0x7f0ec411cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:23.089 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.072+0000 7f0ec359e700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec4131af0 0x7f0ec411cad0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009e50 tx=0x7f0ebc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 1 -- 172.21.15.104:0/2419139817 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc00cd60 con 0x7f0ec4131af0 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: dump: 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:23.090 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 1 -- 172.21.15.104:0/2419139817 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc00f040 con 0x7f0ec4131af0 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 5 monclient: authenticate success, global_id 4182 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 1 -- 172.21.15.104:0/2419139817 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0146d0 con 0x7f0ec4131af0 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 10 monclient: dump: 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:23.091 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 10 monclient: _renew_subs 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac009420 con 0x7f0ec4131af0 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec159a700 4 set_mon_vals no callback set 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 10 monclient: _renew_subs 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:23.092 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec411e6a0 con 0x7f0ec4131af0 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 1 -- 172.21.15.104:0/2419139817 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f0ebc014cd0 con 0x7f0ec4131af0 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ebaffd700 1 -- 172.21.15.104:0/2419139817 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f0ebc01da30 con 0x7f0ec4131af0 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.076+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec4072460 con 0x7f0ec4131af0 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.080+0000 7f0ebaffd700 1 -- 172.21.15.104:0/2419139817 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f0ebc03b990 con 0x7f0ec4131af0 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.080+0000 7f0ebaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:23.093 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.080+0000 7f0ebaffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:23.159 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:23.159 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:23.159 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:23.159 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ec409f350 con 0x7f0ec4131af0 2024-09-18T14:21:23.160 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ebaffd700 1 -- 172.21.15.104:0/2419139817 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f0ebc02a070 con 0x7f0ec4131af0 2024-09-18T14:21:23.160 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ebaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:23.160 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ebaffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:23.160 INFO:teuthology.misc.health.smithi104.stdout: 2024-09-18T14:21:23.160 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":9,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:21:23.113726+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 10 monclient: shutdown 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec4131af0 msgr2=0x7f0ec411cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec4131af0 0x7f0ec411cad0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009e50 tx=0x7f0ebc00b040 comp rx=0 tx=0).stop 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 shutdown_connections 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ec4131af0 0x7f0ec411cad0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ec40a2030 0x7f0ec412f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.162 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 --2- 172.21.15.104:0/2419139817 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0ec409ecc0 0x7f0ec41413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:23.163 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 >> 172.21.15.104:0/2419139817 conn(0x7f0ec40635a0 msgr2=0x7f0ec412fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:23.163 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 shutdown_connections 2024-09-18T14:21:23.163 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:23.152+0000 7f0ec8a5c700 1 -- 172.21.15.104:0/2419139817 wait complete. 2024-09-18T14:21:23.175 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-18T14:21:23.210 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:23.203+0000 7ff37b32f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T14:21:23.378 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:23.371+0000 7ff37b32f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T14:21:23.445 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:23.439+0000 7ff37b32f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T14:21:23.603 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:23.595+0000 7ff37b32f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T14:21:23.670 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:23.663+0000 7ff37b32f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T14:21:24.131 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-18T14:21:24.123+0000 7ff2b123a700 -1 osd.4 0 waiting for initial osdmap 2024-09-18T14:21:24.131 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-18T14:21:24.123+0000 7f2b5efe9700 -1 osd.5 0 waiting for initial osdmap 2024-09-18T14:21:24.132 INFO:tasks.ceph.osd.3.smithi174.stderr:2024-09-18T14:21:24.123+0000 7f0707f8e700 -1 osd.3 0 waiting for initial osdmap 2024-09-18T14:21:24.132 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-09-18T14:21:24.124+0000 7f1a39f6a700 -1 osd.0 0 waiting for initial osdmap 2024-09-18T14:21:24.132 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-09-18T14:21:24.124+0000 7f49a22f6700 -1 osd.1 0 waiting for initial osdmap 2024-09-18T14:21:24.133 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-09-18T14:21:24.124+0000 7fa8c6d1c700 -1 osd.2 0 waiting for initial osdmap 2024-09-18T14:21:24.133 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-18T14:21:24.127+0000 7ff2ad7e4700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:21:24.134 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-18T14:21:24.127+0000 7f2b5ad92700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:21:24.134 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-09-18T14:21:24.124+0000 7f1a36514700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:21:24.135 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-09-18T14:21:24.128+0000 7fa8c32c6700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:21:24.135 INFO:tasks.ceph.osd.3.smithi174.stderr:2024-09-18T14:21:24.127+0000 7f0703d37700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:21:24.135 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-09-18T14:21:24.128+0000 7f499e8a0700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:21:24.382 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:24.375+0000 7ff37b32f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T14:21:25.412 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:25.403+0000 7ff36439f700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T14:21:25.413 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-18T14:21:25.407+0000 7ff36439f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T14:21:29.781 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T14:21:29.885 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient: get_monmap_and_config 2024-09-18T14:21:29.885 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient: build_initial_monmap 2024-09-18T14:21:29.885 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient: monmap: 2024-09-18T14:21:29.885 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-09-18T14:21:29.885 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:29.882080+0000 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:29.882080+0000 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 1 Processor -- start 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 1 -- start start 2024-09-18T14:21:29.886 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient: init 2024-09-18T14:21:29.890 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:29.890 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T14:21:29.890 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8a180a1950 0x7f8a180960c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient(hunting): picked mon.noname-c con 0x7f8a180a1950 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a1809aad0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 0x7f8a1812ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient(hunting): picked mon.noname-b con 0x7f8a1809aea0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1809b300 con 0x7f8a1809aad0 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a18097cd0 con 0x7f8a180a1950 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180a1d20 con 0x7f8a1809aea0 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.876+0000 7f8a1de3b700 10 monclient(hunting): authenticate will time out at 810.871855s 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 0x7f8a1812ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:29.891 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 0x7f8a1812ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:38776/0 (socket says 172.21.15.104:38776) 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 -- 172.21.15.104:0/2732063336 learned_addr learned my addr 172.21.15.104:0/2732063336 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): get_auth_request con 0x7f8a1809aea0 auth_method 0 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 1 --2- 172.21.15.104:0/2732063336 >> v2:172.21.15.174:3300/0 conn(0x7f8a180a1950 0x7f8a180960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 10 monclient(hunting): get_auth_request con 0x7f8a180a1950 auth_method 0 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:29.892 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1ce39700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 -- 172.21.15.104:0/2732063336 >> v2:172.21.15.174:3300/0 conn(0x7f8a180a1950 msgr2=0x7f8a180960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 --2- 172.21.15.104:0/2732063336 >> v2:172.21.15.174:3300/0 conn(0x7f8a180a1950 0x7f8a180960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 -- 172.21.15.104:0/2732063336 >> v1:172.21.15.174:6789/0 conn(0x7f8a1809aad0 legacy=0x7f8a1812e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient: _finish_hunting 0 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 monclient: found mon.noname-b 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:29.893 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 -- 172.21.15.104:0/2732063336 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1809bcd0 con 0x7f8a1809aea0 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 -- 172.21.15.104:0/2732063336 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 msgr2=0x7f8a1812ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1d63a700 1 --2- 172.21.15.104:0/2732063336 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 0x7f8a1812ef60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8a100099b0 tx=0x7f8a10009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1bc1c4082d7df3c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 1 -- 172.21.15.104:0/2732063336 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10009ea0 con 0x7f8a1809aea0 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: dump: 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:29.894 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 5 monclient: authenticate success, global_id 4202 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 1 -- 172.21.15.104:0/2732063336 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1000ba70 con 0x7f8a1809aea0 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 4 set_mon_vals no callback set 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 1 -- 172.21.15.104:0/2732063336 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1000be30 con 0x7f8a1809aea0 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:29.895 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a177fe700 10 monclient: dump: 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 10 monclient: shutdown 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2732063336 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 msgr2=0x7f8a1812ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:29.896 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.880+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2732063336 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 0x7f8a1812ef60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8a100099b0 tx=0x7f8a10009250 comp rx=0 tx=0).stop 2024-09-18T14:21:29.897 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2732063336 shutdown_connections 2024-09-18T14:21:29.897 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2732063336 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aea0 0x7f8a1812ef60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:29.897 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2732063336 >> v2:172.21.15.174:3300/0 conn(0x7f8a180a1950 0x7f8a180960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:29.897 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2732063336 >> 172.21.15.104:0/2732063336 conn(0x7f8a18063cb0 msgr2=0x7f8a180a0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:29.897 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2732063336 shutdown_connections 2024-09-18T14:21:29.897 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2732063336 wait complete. 2024-09-18T14:21:29.901 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient: build_initial_monmap 2024-09-18T14:21:29.901 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient: monmap: 2024-09-18T14:21:29.901 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-09-18T14:21:29.901 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:29.901 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:29.901 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 Processor -- start 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- start start 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient: init 2024-09-18T14:21:29.902 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 0x7f8a1809c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): picked mon.noname-c con 0x7f8a1809aad0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a1809aea0 0x7f8a18130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): picked mon.noname-b con 0x7f8a1809aea0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a180a1950 0x7f8a1813ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): picked mon.noname-a con 0x7f8a180a1950 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1811ed50 con 0x7f8a180a1950 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18131700 con 0x7f8a1809aea0 2024-09-18T14:21:29.903 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1811ea70 con 0x7f8a1809aad0 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1de3b700 10 monclient(hunting): authenticate will time out at 810.878375s 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 0x7f8a1809c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a180a1950 0x7f8a1813ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 0x7f8a1809c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:38790/0 (socket says 172.21.15.104:38790) 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 -- 172.21.15.104:0/2503605527 learned_addr learned my addr 172.21.15.104:0/2503605527 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): get_auth_request con 0x7f8a1809aad0 auth_method 0 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:29.904 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 10 monclient(hunting): get_auth_request con 0x7f8a180a1950 auth_method 0 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a1809aea0 0x7f8a18130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:29.905 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1d63a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 10 monclient(hunting): get_auth_request con 0x7f8a1809aea0 auth_method 0 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 -- 172.21.15.104:0/2503605527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a1809aea0 msgr2=0x7f8a18130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a1809aea0 0x7f8a18130940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:29.906 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 -- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a180a1950 msgr2=0x7f8a1813ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a180a1950 0x7f8a1813ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient: _finish_hunting 0 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 monclient: found mon.noname-c 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.884+0000 7f8a1ce39700 1 -- 172.21.15.104:0/2503605527 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a100094f0 con 0x7f8a1809aad0 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1ce39700 1 -- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 msgr2=0x7f8a1809c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1ce39700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 0x7f8a1809c120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c00ec80 tx=0x7f8a0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:29.907 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 1 -- 172.21.15.104:0/2503605527 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c009690 con 0x7f8a1809aad0 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: dump: 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:29.908 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: _finish_auth 0 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 1 -- 172.21.15.104:0/2503605527 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c00c9e0 con 0x7f8a1809aad0 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 5 monclient: authenticate success, global_id 4205 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 10 monclient: _renew_subs 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:29.909 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1811f030 con 0x7f8a1809aad0 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 1 -- 172.21.15.104:0/2503605527 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c00cda0 con 0x7f8a1809aad0 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 10 monclient: dump: 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:29.910 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr: 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 10 monclient: _renew_subs 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1811ca20 con 0x7f8a1809aad0 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a167fc700 4 set_mon_vals no callback set 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 1 -- 172.21.15.104:0/2503605527 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f8a0c013680 con 0x7f8a1809aad0 2024-09-18T14:21:29.911 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a18072460 con 0x7f8a1809aad0 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 1 --2- 172.21.15.104:0/2503605527 >> v2:172.21.15.174:6812/17603 conn(0x7f8a04035a80 0x7f8a04037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.888+0000 7f8a14ff9700 1 -- 172.21.15.104:0/2503605527 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3149+0+0 (secure 0 0 0) 0x7f8a0c047f70 con 0x7f8a1809aad0 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.892+0000 7f8a17fff700 1 --2- 172.21.15.104:0/2503605527 >> v2:172.21.15.174:6812/17603 conn(0x7f8a04035a80 0x7f8a04037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.892+0000 7f8a17fff700 10 monclient: get_auth_request con 0x7f8a04035a80 auth_method 0 2024-09-18T14:21:29.912 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.892+0000 7f8a17fff700 1 --2- 172.21.15.104:0/2503605527 >> v2:172.21.15.174:6812/17603 conn(0x7f8a04035a80 0x7f8a04037f30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a08007fd0 tx=0x7f8a080026a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:29.913 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.904+0000 7f8a14ff9700 1 -- 172.21.15.104:0/2503605527 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8a0c0123d0 con 0x7f8a1809aad0 2024-09-18T14:21:29.913 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.904+0000 7f8a14ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:29.913 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:29.904+0000 7f8a14ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:30.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.068+0000 7f8a1de3b700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:30.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.068+0000 7f8a1de3b700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:30.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.068+0000 7f8a1de3b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:30.077 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.068+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a18068900 con 0x7f8a1809aad0 2024-09-18T14:21:30.078 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a14ff9700 1 -- 172.21.15.104:0/2503605527 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+8481 (secure 0 0 0) 0x7f8a0c0232f0 con 0x7f8a1809aad0 2024-09-18T14:21:30.078 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a14ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:30.078 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a14ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:30.078 INFO:teuthology.misc.health.smithi104.stdout: 2024-09-18T14:21:30.079 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":15,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:21:29.249319+0000","last_up_change":"2024-09-18T14:21:25.123789+0000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T14:21:26.474399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":17763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":17763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":17763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":17763}]},"public_addr":"172.21.15.104:6808/17763","cluster_addr":"172.21.15.104:6809/17763","heartbeat_back_addr":"172.21.15.104:6811/17763","heartbeat_front_addr":"172.21.15.104:6810/17763","state":["exists","up"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":17761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":17761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":17761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":17761}]},"public_addr":"172.21.15.104:6804/17761","cluster_addr":"172.21.15.104:6805/17761","heartbeat_back_addr":"172.21.15.104:6807/17761","heartbeat_front_addr":"172.21.15.104:6806/17761","state":["exists","up"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":17762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":17762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":17762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":17762}]},"public_addr":"172.21.15.104:6800/17762","cluster_addr":"172.21.15.104:6801/17762","heartbeat_back_addr":"172.21.15.104:6803/17762","heartbeat_front_addr":"172.21.15.104:6802/17762","state":["exists","up"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":17845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":17845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":17845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":17845}]},"public_addr":"172.21.15.174:6808/17845","cluster_addr":"172.21.15.174:6809/17845","heartbeat_back_addr":"172.21.15.174:6811/17845","heartbeat_front_addr":"172.21.15.174:6810/17845","state":["exists","up"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":17847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6801","nonce":17847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":17847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":17847}]},"public_addr":"172.21.15.174:6800/17847","cluster_addr":"172.21.15.174:6801/17847","heartbeat_back_addr":"172.21.15.174:6803/17847","heartbeat_front_addr":"172.21.15.174:6802/17847","state":["exists","up"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":17846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":17846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":17846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":17846}]},"public_addr":"172.21.15.174:6804/17846","cluster_addr":"172.21.15.174:6805/17846","heartbeat_back_addr":"172.21.15.174:6807/17846","heartbeat_front_addr":"172.21.15.174:6806/17846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.237413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.242543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.214389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.213881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.183415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.230911+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:21:30.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 >> v2:172.21.15.174:6812/17603 conn(0x7f8a04035a80 msgr2=0x7f8a04037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:30.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2503605527 >> v2:172.21.15.174:6812/17603 conn(0x7f8a04035a80 0x7f8a04037f30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a08007fd0 tx=0x7f8a080026a0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 10 monclient: shutdown 2024-09-18T14:21:30.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:30.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 msgr2=0x7f8a1809c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:30.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 0x7f8a1809c120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c00ec80 tx=0x7f8a0c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.082 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 shutdown_connections 2024-09-18T14:21:30.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2503605527 >> v2:172.21.15.174:6812/17603 conn(0x7f8a04035a80 0x7f8a04037f30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a180a1950 0x7f8a1813ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a1809aea0 0x7f8a18130940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 --2- 172.21.15.104:0/2503605527 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8a1809aad0 0x7f8a1809c120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 >> 172.21.15.104:0/2503605527 conn(0x7f8a18063cb0 msgr2=0x7f8a1809fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:30.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 shutdown_connections 2024-09-18T14:21:30.083 INFO:teuthology.misc.health.smithi104.stderr:2024-09-18T14:21:30.072+0000 7f8a1de3b700 1 -- 172.21.15.104:0/2503605527 wait complete. 2024-09-18T14:21:30.098 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-18T14:21:30.098 INFO:tasks.ceph:Creating RBD pool 2024-09-18T14:21:30.098 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient: get_monmap_and_config 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient: build_initial_monmap 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient: monmap: 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:30.203873+0000 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:30.203873+0000 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:30.207 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 1 Processor -- start 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 1 -- start start 2024-09-18T14:21:30.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient: init 2024-09-18T14:21:30.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:30.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T14:21:30.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f298809a6b0 0x7f2988131be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:30.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient(hunting): picked mon.noname-c con 0x7f298809a6b0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:30.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f298809aa80 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2988136f10 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298809b6e0 con 0x7f2988136f10 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29880a1280 con 0x7f298809aa80 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f298809b9c0 con 0x7f298809a6b0 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298e323700 10 monclient(hunting): authenticate will time out at 811.193708s 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298db22700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2988136f10 0x7f29881392f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47416/0 (socket says 172.21.15.104:47416) 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298db22700 1 -- 172.21.15.104:0/1661669066 learned_addr learned my addr 172.21.15.104:0/1661669066 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 1 --2- 172.21.15.104:0/1661669066 >> v2:172.21.15.174:3300/0 conn(0x7f298809a6b0 0x7f2988131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): get_auth_request con 0x7f298809a6b0 auth_method 0 2024-09-18T14:21:30.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient(hunting): handle_auth_done global_id 4207 payload 274 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 1 -- 172.21.15.104:0/1661669066 >> v1:172.21.15.174:6789/0 conn(0x7f298809aa80 legacy=0x7f2988132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 1 -- 172.21.15.104:0/1661669066 >> v1:172.21.15.104:6789/0 conn(0x7f2988136f10 legacy=0x7f29881392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient: _finish_hunting 0 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 1 monclient: found mon.noname-c 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:30.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.200+0000 7f298d321700 1 -- 172.21.15.104:0/1661669066 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2988099e60 con 0x7f298809a6b0 2024-09-18T14:21:30.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298d321700 1 -- 172.21.15.104:0/1661669066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f298809a6b0 msgr2=0x7f2988131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:30.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298d321700 1 --2- 172.21.15.104:0/1661669066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f298809a6b0 0x7f2988131be0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f298806aae0 tx=0x7f297c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac73ccfb1b6613f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:30.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 1 -- 172.21.15.104:0/1661669066 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c00b9a0 con 0x7f298809a6b0 2024-09-18T14:21:30.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:30.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: dump: 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:30.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 1 -- 172.21.15.104:0/1661669066 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c00bb20 con 0x7f298809a6b0 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 5 monclient: authenticate success, global_id 4207 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 4 set_mon_vals no callback set 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 1 -- 172.21.15.104:0/1661669066 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c011420 con 0x7f298809a6b0 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f2987fff700 10 monclient: dump: 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 10 monclient: shutdown 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 1 -- 172.21.15.104:0/1661669066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f298809a6b0 msgr2=0x7f2988131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 1 --2- 172.21.15.104:0/1661669066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f298809a6b0 0x7f2988131be0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f298806aae0 tx=0x7f297c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 1 -- 172.21.15.104:0/1661669066 shutdown_connections 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 1 --2- 172.21.15.104:0/1661669066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f298809a6b0 0x7f2988131be0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 1 -- 172.21.15.104:0/1661669066 >> 172.21.15.104:0/1661669066 conn(0x7f29880a53b0 msgr2=0x7f29880a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 1 -- 172.21.15.104:0/1661669066 shutdown_connections 2024-09-18T14:21:30.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 1 -- 172.21.15.104:0/1661669066 wait complete. 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 10 monclient: build_initial_monmap 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.204+0000 7f298e323700 10 monclient: monmap: 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 Processor -- start 2024-09-18T14:21:30.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 -- start start 2024-09-18T14:21:30.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient: init 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 0x7f298806f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): picked mon.noname-c con 0x7f298809a6b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f298809aa80 0x7f298806adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): picked mon.noname-a con 0x7f298809aa80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2988136f10 0x7f298806b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): picked mon.noname-b con 0x7f2988136f10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298809aee0 con 0x7f298809aa80 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2988136bb0 con 0x7f2988136f10 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298806dc50 con 0x7f298809a6b0 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298e323700 10 monclient(hunting): authenticate will time out at 811.201217s 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 0x7f298806f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f298809aa80 0x7f298806adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 0x7f298806f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35626/0 (socket says 172.21.15.104:35626) 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 -- 172.21.15.104:0/993894099 learned_addr learned my addr 172.21.15.104:0/993894099 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): get_auth_request con 0x7f298809a6b0 auth_method 0 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 10 monclient(hunting): get_auth_request con 0x7f298809aa80 auth_method 0 2024-09-18T14:21:30.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2988136f10 0x7f298806b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 10 monclient(hunting): get_auth_request con 0x7f2988136f10 auth_method 0 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:30.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298db22700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient(hunting): handle_auth_done global_id 4208 payload 824 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 -- 172.21.15.104:0/993894099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2988136f10 msgr2=0x7f298806b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2988136f10 0x7f298806b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 -- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f298809aa80 msgr2=0x7f298806adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f298809aa80 0x7f298806adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient: _finish_hunting 0 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 monclient: found mon.noname-c 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298d321700 1 -- 172.21.15.104:0/993894099 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c009420 con 0x7f298809a6b0 2024-09-18T14:21:30.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.208+0000 7f298cb20700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f298809aa80 0x7f298806adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298d321700 1 -- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 msgr2=0x7f298806f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298d321700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 0x7f298806f8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f29880a5310 tx=0x7f297c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 1 -- 172.21.15.104:0/993894099 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c006b30 con 0x7f298809a6b0 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: dump: 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:30.228 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: _finish_auth 0 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 1 -- 172.21.15.104:0/993894099 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c00bea0 con 0x7f298809a6b0 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 5 monclient: authenticate success, global_id 4208 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 10 monclient: _renew_subs 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298806df30 con 0x7f298809a6b0 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 1 -- 172.21.15.104:0/993894099 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c01a6b0 con 0x7f298809a6b0 2024-09-18T14:21:30.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 10 monclient: dump: 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29877fe700 4 set_mon_vals no callback set 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 10 monclient: _renew_subs 2024-09-18T14:21:30.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298806e910 con 0x7f298809a6b0 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298806e910 con 0x7f298809a6b0 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 1 -- 172.21.15.104:0/993894099 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f297c01aa60 con 0x7f298809a6b0 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 1 --2- 172.21.15.104:0/993894099 >> v2:172.21.15.174:6812/17603 conn(0x7f2974035a80 0x7f2974037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f29857fa700 1 -- 172.21.15.104:0/993894099 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3149+0+0 (secure 0 0 0) 0x7f297c049d80 con 0x7f298809a6b0 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298cb20700 1 --2- 172.21.15.104:0/993894099 >> v2:172.21.15.174:6812/17603 conn(0x7f2974035a80 0x7f2974037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:30.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.212+0000 7f298cb20700 10 monclient: get_auth_request con 0x7f2974035a80 auth_method 0 2024-09-18T14:21:30.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.216+0000 7f298cb20700 1 --2- 172.21.15.104:0/993894099 >> v2:172.21.15.174:6812/17603 conn(0x7f2974035a80 0x7f2974037f30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f298809b110 tx=0x7f2978007ea0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:30.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.224+0000 7f29857fa700 1 -- 172.21.15.104:0/993894099 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f297c0243e0 con 0x7f298809a6b0 2024-09-18T14:21:30.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.224+0000 7f29857fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:30.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.224+0000 7f29857fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:30.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.388+0000 7f298e323700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T14:21:30.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.388+0000 7f298e323700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T14:21:30.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.388+0000 7f298e323700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:30.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:30.388+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f298806e910 con 0x7f298809a6b0 2024-09-18T14:21:31.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.268+0000 7f29857fa700 1 -- 172.21.15.104:0/993894099 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7f297c02d390 con 0x7f298809a6b0 2024-09-18T14:21:31.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.268+0000 7f29857fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T14:21:31.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.268+0000 7f29857fa700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-18T14:21:31.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 >> v2:172.21.15.174:6812/17603 conn(0x7f2974035a80 msgr2=0x7f2974037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:31.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 --2- 172.21.15.104:0/993894099 >> v2:172.21.15.174:6812/17603 conn(0x7f2974035a80 0x7f2974037f30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f298809b110 tx=0x7f2978007ea0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 10 monclient: shutdown 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 msgr2=0x7f298806f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 0x7f298806f8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f29880a5310 tx=0x7f297c016000 comp rx=0 tx=0).stop 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 shutdown_connections 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 --2- 172.21.15.104:0/993894099 >> v2:172.21.15.174:6812/17603 conn(0x7f2974035a80 0x7f2974037f30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2988136f10 0x7f298806b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f298809aa80 0x7f298806adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 --2- 172.21.15.104:0/993894099 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f298809a6b0 0x7f298806f8d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 >> 172.21.15.104:0/993894099 conn(0x7f29880a53b0 msgr2=0x7f298809cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 shutdown_connections 2024-09-18T14:21:31.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.276+0000 7f298e323700 1 -- 172.21.15.104:0/993894099 wait complete. 2024-09-18T14:21:31.284 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'rbd' created 2024-09-18T14:21:31.295 DEBUG:teuthology.orchestra.run.smithi104:> rbd --cluster ceph pool init rbd 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient: get_monmap_and_config 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient: build_initial_monmap 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient: monmap: 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:31.329001+0000 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:31.329001+0000 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:31.332 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:31.333 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:31.333 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:31.333 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:31.333 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:31.333 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 Processor -- start 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 -- start start 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient: init 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient: _reopen_session rank -1 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 --2- >> v2:172.21.15.104:3300/0 conn(0x55e528522f40 0x55e528528ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): picked mon.noname-a con 0x55e528522f40 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 --2- >> v2:172.21.15.174:3300/0 conn(0x55e528529280 0x55e52852a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): picked mon.noname-c con 0x55e528529280 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:31.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 0x55e52852a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): picked mon.noname-b con 0x55e528529650 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x55e52847f360 con 0x55e528522f40 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x55e528413bc0 con 0x55e528529280 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e52843fd10 con 0x55e528529650 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): _renew_subs 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700f81e4c0 10 monclient(hunting): authenticate will time out at 812.319331s 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 0x55e52852a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 0x55e52852a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35634/0 (socket says 172.21.15.104:35634) 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 1 -- 172.21.15.104:0/2132073377 learned_addr learned my addr 172.21.15.104:0/2132073377 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:31.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 10 monclient(hunting): get_auth_request con 0x55e528529650 auth_method 0 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.324+0000 7f700e354700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700db53700 1 --2- 172.21.15.104:0/2132073377 >> v2:172.21.15.104:3300/0 conn(0x55e528522f40 0x55e528528ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700d352700 1 --2- 172.21.15.104:0/2132073377 >> v2:172.21.15.174:3300/0 conn(0x55e528529280 0x55e52852a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700d352700 10 monclient(hunting): get_auth_request con 0x55e528529280 auth_method 0 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700d352700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700d352700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:31.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700d352700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700db53700 10 monclient(hunting): get_auth_request con 0x55e528522f40 auth_method 0 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700db53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700db53700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700db53700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 -- 172.21.15.104:0/2132073377 >> v2:172.21.15.174:3300/0 conn(0x55e528529280 msgr2=0x55e52852a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 --2- 172.21.15.104:0/2132073377 >> v2:172.21.15.174:3300/0 conn(0x55e528529280 0x55e52852a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 -- 172.21.15.104:0/2132073377 >> v2:172.21.15.104:3300/0 conn(0x55e528522f40 msgr2=0x55e528528ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 --2- 172.21.15.104:0/2132073377 >> v2:172.21.15.104:3300/0 conn(0x55e528522f40 0x55e528528ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 10 monclient: _finish_hunting 0 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 monclient: found mon.noname-b 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:31.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 -- 172.21.15.104:0/2132073377 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e528448660 con 0x55e528529650 2024-09-18T14:21:31.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 -- 172.21.15.104:0/2132073377 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 msgr2=0x55e52852a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:31.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700e354700 1 --2- 172.21.15.104:0/2132073377 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 0x55e52852a700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x55e528526e40 tx=0x7f7000005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4844720f6a2837b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:31.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 1 -- 172.21.15.104:0/2132073377 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700000e720 con 0x55e528529650 2024-09-18T14:21:31.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:31.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: dump: 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: _finish_auth 0 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:31.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700f81e4c0 5 monclient: authenticate success, global_id 4211 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700f81e4c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 1 -- 172.21.15.104:0/2132073377 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700000e880 con 0x55e528529650 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 1 -- 172.21.15.104:0/2132073377 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700000ec20 con 0x55e528529650 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700f81e4c0 10 monclient: get_monmap_and_config success 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700f81e4c0 4 set_mon_vals no callback set 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.328+0000 7f700cb51700 10 monclient: dump: 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:31.342 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 10 monclient: shutdown 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 -- 172.21.15.104:0/2132073377 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 msgr2=0x55e52852a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/2132073377 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 0x55e52852a700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x55e528526e40 tx=0x7f7000005e00 comp rx=0 tx=0).stop 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 -- 172.21.15.104:0/2132073377 shutdown_connections 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/2132073377 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529650 0x55e52852a700 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/2132073377 >> v2:172.21.15.174:3300/0 conn(0x55e528529280 0x55e52852a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/2132073377 >> v2:172.21.15.104:3300/0 conn(0x55e528522f40 0x55e528528ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 -- 172.21.15.104:0/2132073377 >> 172.21.15.104:0/2132073377 conn(0x55e52852c2c0 msgr2=0x55e52852c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 -- 172.21.15.104:0/2132073377 shutdown_connections 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 -- 172.21.15.104:0/2132073377 wait complete. 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 10 monclient: build_initial_monmap 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 10 monclient: monmap: 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:31.345 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 Processor -- start 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 1 -- start start 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.332+0000 7f700f81e4c0 10 monclient: init 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient: _reopen_session rank -1 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529280 0x55e5285cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): picked mon.noname-c con 0x55e528529280 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e528529650 0x55e5285c2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): picked mon.noname-b con 0x55e528529650 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e5285c2860 0x55e5285a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): picked mon.noname-a con 0x55e5285c2860 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e528422f40 con 0x55e5285c2860 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e528416680 con 0x55e528529650 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5285aa7d0 con 0x55e528529280 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): _renew_subs 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700f81e4c0 10 monclient(hunting): authenticate will time out at 812.328437s 2024-09-18T14:21:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529280 0x55e5285cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e5285c2860 0x55e5285a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529280 0x55e5285cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35648/0 (socket says 172.21.15.104:35648) 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 -- 172.21.15.104:0/988544493 learned_addr learned my addr 172.21.15.104:0/988544493 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): get_auth_request con 0x55e528529280 auth_method 0 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 10 monclient(hunting): get_auth_request con 0x55e5285c2860 auth_method 0 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 1 --2- 172.21.15.104:0/988544493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e528529650 0x55e5285c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 10 monclient(hunting): get_auth_request con 0x55e528529650 auth_method 0 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700d352700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 -- 172.21.15.104:0/988544493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e528529650 msgr2=0x55e5285c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 --2- 172.21.15.104:0/988544493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e528529650 0x55e5285c2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 -- 172.21.15.104:0/988544493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e5285c2860 msgr2=0x55e5285a6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 --2- 172.21.15.104:0/988544493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e5285c2860 0x55e5285a6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient: _finish_hunting 0 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 monclient: found mon.noname-c 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700000a180 con 0x55e528529280 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700e354700 1 --2- 172.21.15.104:0/988544493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e5285c2860 0x55e5285a6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 -- 172.21.15.104:0/988544493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529280 msgr2=0x55e5285cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.336+0000 7f700db53700 1 --2- 172.21.15.104:0/988544493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529280 0x55e5285cb420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7004008830 tx=0x7f7004005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700400f070 con 0x55e528529280 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: dump: 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:31.354 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 5 monclient: authenticate success, global_id 4214 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 10 monclient: _renew_subs 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7004006740 con 0x55e528529280 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e5284217f0 con 0x55e528529280 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004006ae0 con 0x55e528529280 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 10 monclient: dump: 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffffff700 4 set_mon_vals no callback set 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 10 monclient: _renew_subs 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e52842e660 con 0x55e528529280 2024-09-18T14:21:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f7004013070 con 0x55e528529280 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6812/17603 conn(0x7f6fec034320 0x7f6fec0367d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3491+0+0 (secure 0 0 0) 0x7f70040464a0 con 0x55e528529280 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x55e528421b80 con 0x55e528529280 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700d352700 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6812/17603 conn(0x7f6fec034320 0x7f6fec0367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:31.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.340+0000 7f700d352700 10 monclient: get_auth_request con 0x7f6fec034320 auth_method 0 2024-09-18T14:21:31.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:31.344+0000 7f700d352700 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6812/17603 conn(0x7f6fec034320 0x7f6fec0367d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x55e52850c4a0 tx=0x7f6ff8005bb0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:32.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.280+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 141+0+0 (secure 0 0 0) 0x7f700401a020 con 0x55e528529280 2024-09-18T14:21:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.280+0000 7f6ffdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T14:21:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.280+0000 7f6ffdffb700 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-18T14:21:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.280+0000 7f700f81e4c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:32.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.280+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55e528422d30 con 0x55e528529280 2024-09-18T14:21:32.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7004014e70 con 0x55e528529280 2024-09-18T14:21:32.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f6ffdffb700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T14:21:32.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f6ffffff700 10 monclient: _renew_subs 2024-09-18T14:21:32.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f6ffffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:32.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f6ffffff700 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=17}) v3 -- 0x7f6fe8002990 con 0x55e528529280 2024-09-18T14:21:32.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 650+0+0 (secure 0 0 0) 0x7f7004013320 con 0x55e528529280 2024-09-18T14:21:32.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6804/17846 conn(0x55e5285d27a0 0x55e5285d4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:32.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 --> v2:172.21.15.174:6804/17846 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) v8 -- 0x55e5285d50c0 con 0x55e5285d27a0 2024-09-18T14:21:32.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f700e354700 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6804/17846 conn(0x55e5285d27a0 0x55e5285d4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:32.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.288+0000 7f700e354700 10 monclient: get_auth_request con 0x55e5285d27a0 auth_method 0 2024-09-18T14:21:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.292+0000 7f700e354700 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6804/17846 conn(0x55e5285d27a0 0x55e5285d4b80 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.296+0000 7f700e354700 1 -- 172.21.15.104:0/988544493 <== osd.5 v2:172.21.15.174:6804/17846 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f700000ea80 con 0x55e5285d27a0 2024-09-18T14:21:32.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 --> v2:172.21.15.174:6804/17846 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) v8 -- 0x55e528622660 con 0x55e5285d27a0 2024-09-18T14:21:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.300+0000 7f700e354700 1 -- 172.21.15.104:0/988544493 <== osd.5 v2:172.21.15.174:6804/17846 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f700000ea80 con 0x55e5285d27a0 2024-09-18T14:21:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.300+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 --> v2:172.21.15.174:6804/17846 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) v8 -- 0x55e528621740 con 0x55e5285d27a0 2024-09-18T14:21:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.300+0000 7f700e354700 1 -- 172.21.15.104:0/988544493 <== osd.5 v2:172.21.15.174:6804/17846 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f700000ea80 con 0x55e5285d27a0 2024-09-18T14:21:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.300+0000 7f6fff7fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:32.300+0000 7f6fff7fe700 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f6fe0001ee0 con 0x55e528529280 2024-09-18T14:21:33.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.276+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v18) v1 ==== 55+0+0 (secure 0 0 0) 0x7f7004011500 con 0x55e528529280 2024-09-18T14:21:33.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.276+0000 7f6ffdffb700 10 monclient: _renew_subs 2024-09-18T14:21:33.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.276+0000 7f6ffdffb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:33.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.276+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=18}) v3 -- 0x7f6fec03a1c0 con 0x55e528529280 2024-09-18T14:21:33.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.276+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 10 ==== osd_map(18..18 src has 1..18) v4 ==== 618+0+0 (secure 0 0 0) 0x7f700401ec90 con 0x55e528529280 2024-09-18T14:21:33.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.276+0000 7f6ffffff700 1 -- 172.21.15.104:0/988544493 --> v2:172.21.15.174:6804/17846 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x7f6fe8005490 con 0x55e5285d27a0 2024-09-18T14:21:33.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.280+0000 7f700e354700 1 -- 172.21.15.104:0/988544493 <== osd.5 v2:172.21.15.174:6804/17846 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f700000ea80 con 0x55e5285d27a0 2024-09-18T14:21:33.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.280+0000 7f6fff7fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:33.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:33.280+0000 7f6fff7fe700 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) v4 -- 0x7f6fe00033b0 con 0x55e528529280 2024-09-18T14:21:34.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.292+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v19) v1 ==== 43+0+0 (secure 0 0 0) 0x7f700401a020 con 0x55e528529280 2024-09-18T14:21:34.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.292+0000 7f6ffdffb700 10 monclient: _renew_subs 2024-09-18T14:21:34.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.292+0000 7f6ffdffb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.292+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=19}) v3 -- 0x7f6fec03a350 con 0x55e528529280 2024-09-18T14:21:34.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.292+0000 7f6ffdffb700 1 -- 172.21.15.104:0/988544493 <== mon.2 v2:172.21.15.104:3301/0 12 ==== osd_map(19..19 src has 1..19) v4 ==== 646+0+0 (secure 0 0 0) 0x7f70040136a0 con 0x55e528529280 2024-09-18T14:21:34.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.292+0000 7f6ffffff700 1 -- 172.21.15.104:0/988544493 --> v2:172.21.15.174:6804/17846 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x7f6fe80058c0 con 0x55e5285d27a0 2024-09-18T14:21:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700e354700 1 -- 172.21.15.104:0/988544493 <== osd.5 v2:172.21.15.174:6804/17846 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f700000ea80 con 0x55e5285d27a0 2024-09-18T14:21:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6804/17846 conn(0x55e5285d27a0 msgr2=0x55e5285d4b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6804/17846 conn(0x55e5285d27a0 0x55e5285d4b80 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6812/17603 conn(0x7f6fec034320 msgr2=0x7f6fec0367d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/988544493 >> v2:172.21.15.174:6812/17603 conn(0x7f6fec034320 0x7f6fec0367d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x55e52850c4a0 tx=0x7f6ff8005bb0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 10 monclient: shutdown 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529280 msgr2=0x55e5285cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 --2- 172.21.15.104:0/988544493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x55e528529280 0x55e5285cb420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7004008830 tx=0x7f7004005c80 comp rx=0 tx=0).stop 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700e354700 1 -- 172.21.15.104:0/988544493 reap_dead start 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 shutdown_connections 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 >> 172.21.15.104:0/988544493 conn(0x55e52852c2c0 msgr2=0x55e52852e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 shutdown_connections 2024-09-18T14:21:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.296+0000 7f700f81e4c0 1 -- 172.21.15.104:0/988544493 wait complete. 2024-09-18T14:21:34.309 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T14:21:34.309 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T14:21:34.309 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T14:21:34.309 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T14:21:34.310 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient: get_monmap_and_config 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient: build_initial_monmap 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient: monmap: 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:34.416315+0000 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:34.416315+0000 2024-09-18T14:21:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:34.420 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.420 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:34.420 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:34.420 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:34.420 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:34.420 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:34.420 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 1 Processor -- start 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 1 -- start start 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient: init 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb009a5b0 0x7faeb009ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient(hunting): picked mon.noname-b con 0x7faeb009a5b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faeb012f980 0x7faeb012fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient(hunting): picked mon.noname-c con 0x7faeb012f980 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faeb0130290 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb0097d10 con 0x7faeb0130290 2024-09-18T14:21:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb0098470 con 0x7faeb012f980 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb0099650 con 0x7faeb009a5b0 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb5bd6700 10 monclient(hunting): authenticate will time out at 815.405882s 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb009a5b0 0x7faeb009ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb53d5700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faeb0130290 0x7faeb012c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47428/0 (socket says 172.21.15.104:47428) 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb53d5700 1 -- 172.21.15.104:0/2104391484 learned_addr learned my addr 172.21.15.104:0/2104391484 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): get_auth_request con 0x7faeb009a5b0 auth_method 0 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 1 --2- 172.21.15.104:0/2104391484 >> v2:172.21.15.174:3300/0 conn(0x7faeb012f980 0x7faeb012fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 10 monclient(hunting): get_auth_request con 0x7faeb012f980 auth_method 0 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faea7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 1 -- 172.21.15.104:0/2104391484 >> v2:172.21.15.174:3300/0 conn(0x7faeb012f980 msgr2=0x7faeb012fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 1 --2- 172.21.15.104:0/2104391484 >> v2:172.21.15.174:3300/0 conn(0x7faeb012f980 0x7faeb012fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 1 -- 172.21.15.104:0/2104391484 >> v1:172.21.15.104:6789/0 conn(0x7faeb0130290 legacy=0x7faeb012c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient: _finish_hunting 0 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 1 monclient: found mon.noname-b 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.412+0000 7faeb4bd4700 1 -- 172.21.15.104:0/2104391484 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb0099930 con 0x7faeb009a5b0 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb4bd4700 1 -- 172.21.15.104:0/2104391484 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb009a5b0 msgr2=0x7faeb009ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb4bd4700 1 --2- 172.21.15.104:0/2104391484 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb009a5b0 0x7faeb009ae20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faea00098e0 tx=0x7faea00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e52d18fc52d5ea0e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 1 -- 172.21.15.104:0/2104391484 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea000b850 con 0x7faeb009a5b0 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: dump: 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.430 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 5 monclient: authenticate success, global_id 4217 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 1 -- 172.21.15.104:0/2104391484 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea000b9b0 con 0x7faeb009a5b0 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 4 set_mon_vals no callback set 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 1 -- 172.21.15.104:0/2104391484 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea000bd50 con 0x7faeb009a5b0 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faea77fe700 10 monclient: dump: 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 10 monclient: shutdown 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 -- 172.21.15.104:0/2104391484 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb009a5b0 msgr2=0x7faeb009ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/2104391484 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb009a5b0 0x7faeb009ae20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faea00098e0 tx=0x7faea00091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 -- 172.21.15.104:0/2104391484 shutdown_connections 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/2104391484 >> v2:172.21.15.174:3300/0 conn(0x7faeb012f980 0x7faeb012fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/2104391484 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb009a5b0 0x7faeb009ae20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 -- 172.21.15.104:0/2104391484 >> 172.21.15.104:0/2104391484 conn(0x7faeb00a12c0 msgr2=0x7faeb00a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 -- 172.21.15.104:0/2104391484 shutdown_connections 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 -- 172.21.15.104:0/2104391484 wait complete. 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 10 monclient: build_initial_monmap 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 10 monclient: monmap: 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:34.434 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 Processor -- start 2024-09-18T14:21:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 1 -- start start 2024-09-18T14:21:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.416+0000 7faeb5bd6700 10 monclient: init 2024-09-18T14:21:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb012f980 0x7faeb011ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): picked mon.noname-a con 0x7faeb012f980 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb0130290 0x7faeb011b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): picked mon.noname-c con 0x7faeb0130290 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb011b700 0x7faeb011bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): picked mon.noname-b con 0x7faeb011b700 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb011e900 con 0x7faeb012f980 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb011cd00 con 0x7faeb011b700 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb011c0b0 con 0x7faeb0130290 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb5bd6700 10 monclient(hunting): authenticate will time out at 815.412521s 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb0130290 0x7faeb011b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb012f980 0x7faeb011ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb012f980 0x7faeb011ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50778/0 (socket says 172.21.15.104:50778) 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 1 -- 172.21.15.104:0/1944833743 learned_addr learned my addr 172.21.15.104:0/1944833743 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): get_auth_request con 0x7faeb0130290 auth_method 0 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 10 monclient(hunting): get_auth_request con 0x7faeb012f980 auth_method 0 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb011b700 0x7faeb011bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 10 monclient(hunting): get_auth_request con 0x7faeb011b700 auth_method 0 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb4bd4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faeb53d5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 -- 172.21.15.104:0/1944833743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb011b700 msgr2=0x7faeb011bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb011b700 0x7faeb011bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 -- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb012f980 msgr2=0x7faeb011ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb012f980 0x7faeb011ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 monclient: found mon.noname-c 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 -- 172.21.15.104:0/1944833743 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0009420 con 0x7faeb0130290 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 -- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb0130290 msgr2=0x7faeb011b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.420+0000 7faea7fff700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb0130290 0x7faeb011b1c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faeac00ea00 tx=0x7faeac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 1 -- 172.21.15.104:0/1944833743 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac013890 con 0x7faeb0130290 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: dump: 2024-09-18T14:21:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: _finish_auth 0 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 1 -- 172.21.15.104:0/1944833743 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac013a10 con 0x7faeb0130290 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 5 monclient: authenticate success, global_id 4220 2024-09-18T14:21:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 10 monclient: _renew_subs 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb00989a0 con 0x7faeb0130290 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 1 -- 172.21.15.104:0/1944833743 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac013dd0 con 0x7faeb0130290 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 10 monclient: dump: 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea6ffd700 4 set_mon_vals no callback set 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 10 monclient: _renew_subs 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb011e240 con 0x7faeb0130290 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 1 -- 172.21.15.104:0/1944833743 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7faeac00c6a0 con 0x7faeb0130290 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 1 --2- 172.21.15.104:0/1944833743 >> v2:172.21.15.174:6812/17603 conn(0x7fae9c035ab0 0x7fae9c037f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb00633b0 con 0x7faeb0130290 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faea4ff9700 1 -- 172.21.15.104:0/1944833743 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7faeac047e10 con 0x7faeb0130290 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb4bd4700 1 --2- 172.21.15.104:0/1944833743 >> v2:172.21.15.174:6812/17603 conn(0x7fae9c035ab0 0x7fae9c037f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.424+0000 7faeb4bd4700 10 monclient: get_auth_request con 0x7fae9c035ab0 auth_method 0 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.428+0000 7faeb4bd4700 1 --2- 172.21.15.104:0/1944833743 >> v2:172.21.15.174:6812/17603 conn(0x7fae9c035ab0 0x7fae9c037f60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faea00030c0 tx=0x7faea0016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.436+0000 7faea4ff9700 1 -- 172.21.15.104:0/1944833743 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7faeac00ce70 con 0x7faeb0130290 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.436+0000 7faea4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:34.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.436+0000 7faea4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:34.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.636+0000 7faeb5bd6700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-18T14:21:34.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.636+0000 7faeb5bd6700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T14:21:34.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.636+0000 7faeb5bd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.636+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7faeb012c2b0 con 0x7faeb0130290 2024-09-18T14:21:34.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.640+0000 7faea4ff9700 1 -- 172.21.15.104:0/1944833743 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+83350 (secure 0 0 0) 0x7faeac0192c0 con 0x7faeb0130290 2024-09-18T14:21:34.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.640+0000 7faea4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T14:21:34.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.640+0000 7faea4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 >> v2:172.21.15.174:6812/17603 conn(0x7fae9c035ab0 msgr2=0x7fae9c037f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/1944833743 >> v2:172.21.15.174:6812/17603 conn(0x7fae9c035ab0 0x7fae9c037f60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faea00030c0 tx=0x7faea0016000 comp rx=0 tx=0).stop 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 10 monclient: shutdown 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb0130290 msgr2=0x7faeb011b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb0130290 0x7faeb011b1c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faeac00ea00 tx=0x7faeac0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 shutdown_connections 2024-09-18T14:21:34.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/1944833743 >> v2:172.21.15.174:6812/17603 conn(0x7fae9c035ab0 0x7fae9c037f60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb011b700 0x7faeb011bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faeb0130290 0x7faeb011b1c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 --2- 172.21.15.104:0/1944833743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb012f980 0x7faeb011ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 >> 172.21.15.104:0/1944833743 conn(0x7faeb00a12c0 msgr2=0x7faeb009f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:34.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 shutdown_connections 2024-09-18T14:21:34.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.644+0000 7faeb5bd6700 1 -- 172.21.15.104:0/1944833743 wait complete. 2024-09-18T14:21:34.665 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":5,"flags":0,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":17603}]},"active_addr":"172.21.15.174:6812/17603","active_change":"2024-09-18T14:21:24.382660+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":1456278727}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":2032070632}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":1312572104}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":2678143315}]}]} 2024-09-18T14:21:34.667 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T14:21:34.668 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T14:21:34.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:21:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient: get_monmap_and_config 2024-09-18T14:21:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient: build_initial_monmap 2024-09-18T14:21:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient: monmap: 2024-09-18T14:21:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:34.767969+0000 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:34.767969+0000 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 1 Processor -- start 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 1 -- start start 2024-09-18T14:21:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient: init 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa78809ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient(hunting): picked mon.noname-b con 0x7fa78809a5b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa78812f980 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa788130460 0x7fa788136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient(hunting): picked mon.noname-a con 0x7fa788130460 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa788097d10 con 0x7fa78812f980 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa788098470 con 0x7fa788130460 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa788099650 con 0x7fa78809a5b0 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78f44e700 10 monclient(hunting): authenticate will time out at 815.757533s 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa788130460 0x7fa788136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa78809ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa788130460 0x7fa788136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50782/0 (socket says 172.21.15.104:50782) 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa78809ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35670/0 (socket says 172.21.15.104:35670) 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 -- 172.21.15.104:0/1480438872 learned_addr learned my addr 172.21.15.104:0/1480438872 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): get_auth_request con 0x7fa788130460 auth_method 0 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 10 monclient(hunting): get_auth_request con 0x7fa78809a5b0 auth_method 0 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78e44c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 -- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 msgr2=0x7fa78809ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 --2- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa78809ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 -- 172.21.15.104:0/1480438872 >> v1:172.21.15.174:6789/0 conn(0x7fa78812f980 legacy=0x7fa78812fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient: _finish_hunting 0 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 monclient: found mon.noname-a 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:34.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.764+0000 7fa78ec4d700 1 -- 172.21.15.104:0/1480438872 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788099930 con 0x7fa788130460 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78ec4d700 1 -- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 msgr2=0x7fa788136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78ec4d700 1 --2- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 0x7fa788136ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa780009f20 tx=0x7fa780009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0288e3a8c83bde2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 1 -- 172.21.15.104:0/1480438872 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78000e070 con 0x7fa788130460 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: dump: 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: _finish_auth 0 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 5 monclient: authenticate success, global_id 4224 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 1 -- 172.21.15.104:0/1480438872 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78000bbe0 con 0x7fa788130460 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 1 -- 172.21.15.104:0/1480438872 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800092e0 con 0x7fa788130460 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78d44a700 10 monclient: dump: 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 4 set_mon_vals no callback set 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 10 monclient: shutdown 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 -- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 msgr2=0x7fa788136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 0x7fa788136ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa780009f20 tx=0x7fa780009840 comp rx=0 tx=0).stop 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 -- 172.21.15.104:0/1480438872 shutdown_connections 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 0x7fa788136ed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1480438872 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa78809ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 -- 172.21.15.104:0/1480438872 >> 172.21.15.104:0/1480438872 conn(0x7fa7880a12c0 msgr2=0x7fa7880a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 -- 172.21.15.104:0/1480438872 shutdown_connections 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 1 -- 172.21.15.104:0/1480438872 wait complete. 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 10 monclient: build_initial_monmap 2024-09-18T14:21:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.768+0000 7fa78f44e700 10 monclient: monmap: 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 Processor -- start 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 -- start start 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient: init 2024-09-18T14:21:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa788136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): picked mon.noname-c con 0x7fa78809a5b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa78812f980 0x7fa7881329a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): picked mon.noname-b con 0x7fa78812f980 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 0x7fa788132ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): picked mon.noname-a con 0x7fa788130460 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7881371a0 con 0x7fa788130460 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78812f4f0 con 0x7fa78812f980 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880989a0 con 0x7fa78809a5b0 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78f44e700 10 monclient(hunting): authenticate will time out at 815.765150s 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa788136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa788136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35684/0 (socket says 172.21.15.104:35684) 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 -- 172.21.15.104:0/1290198715 learned_addr learned my addr 172.21.15.104:0/1290198715 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 0x7fa788132ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): get_auth_request con 0x7fa78809a5b0 auth_method 0 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 10 monclient(hunting): get_auth_request con 0x7fa788130460 auth_method 0 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa78812f980 0x7fa7881329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 10 monclient(hunting): get_auth_request con 0x7fa78812f980 auth_method 0 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78ec4d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78dc4b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 -- 172.21.15.104:0/1290198715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa78812f980 msgr2=0x7fa7881329a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa78812f980 0x7fa7881329a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 -- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 msgr2=0x7fa788132ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 0x7fa788132ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 monclient: found mon.noname-c 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.772+0000 7fa78e44c700 1 -- 172.21.15.104:0/1290198715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa780009ae0 con 0x7fa78809a5b0 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78e44c700 1 -- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 msgr2=0x7fa788136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78e44c700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa788136c60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa78400ec80 tx=0x7fa7840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 1 -- 172.21.15.104:0/1290198715 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78400ca60 con 0x7fa78809a5b0 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: dump: 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 5 monclient: authenticate success, global_id 4226 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 10 monclient: _renew_subs 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 1 -- 172.21.15.104:0/1290198715 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78400cbe0 con 0x7fa78809a5b0 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa788136280 con 0x7fa78809a5b0 2024-09-18T14:21:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 1 -- 172.21.15.104:0/1290198715 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784013470 con 0x7fa78809a5b0 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 10 monclient: dump: 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 10 monclient: _renew_subs 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78cc49700 4 set_mon_vals no callback set 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa788135fa0 con 0x7fa78809a5b0 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 1 -- 172.21.15.104:0/1290198715 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fa784013820 con 0x7fa78809a5b0 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7881368a0 con 0x7fa78809a5b0 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 1 --2- 172.21.15.104:0/1290198715 >> v2:172.21.15.174:6812/17603 conn(0x7fa76c035ab0 0x7fa76c037f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa77e7fc700 1 -- 172.21.15.104:0/1290198715 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fa784048260 con 0x7fa78809a5b0 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78dc4b700 1 --2- 172.21.15.104:0/1290198715 >> v2:172.21.15.174:6812/17603 conn(0x7fa76c035ab0 0x7fa76c037f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.776+0000 7fa78dc4b700 10 monclient: get_auth_request con 0x7fa76c035ab0 auth_method 0 2024-09-18T14:21:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.780+0000 7fa78dc4b700 1 --2- 172.21.15.104:0/1290198715 >> v2:172.21.15.174:6812/17603 conn(0x7fa76c035ab0 0x7fa76c037f60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa788064f90 tx=0x7fa778007ea0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.784+0000 7fa77e7fc700 1 -- 172.21.15.104:0/1290198715 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa7840236b0 con 0x7fa78809a5b0 2024-09-18T14:21:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.784+0000 7fa77e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.784+0000 7fa77e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:34.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.948+0000 7fa78f44e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:34.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.948+0000 7fa78f44e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:34.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.948+0000 7fa78f44e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:34.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.948+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa788136440 con 0x7fa78809a5b0 2024-09-18T14:21:34.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.948+0000 7fa77e7fc700 1 -- 172.21.15.104:0/1290198715 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+10108 (secure 0 0 0) 0x7fa784021070 con 0x7fa78809a5b0 2024-09-18T14:21:34.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.948+0000 7fa77e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:34.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.948+0000 7fa77e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:34.958 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:34.958 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":19,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:21:34.289999+0000","last_up_change":"2024-09-18T14:21:25.123789+0000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T14:21:26.474399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T14:21:30.433492+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":17763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":17763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":17763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":17763}]},"public_addr":"172.21.15.104:6808/17763","cluster_addr":"172.21.15.104:6809/17763","heartbeat_back_addr":"172.21.15.104:6811/17763","heartbeat_front_addr":"172.21.15.104:6810/17763","state":["exists","up"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":17761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":17761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":17761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":17761}]},"public_addr":"172.21.15.104:6804/17761","cluster_addr":"172.21.15.104:6805/17761","heartbeat_back_addr":"172.21.15.104:6807/17761","heartbeat_front_addr":"172.21.15.104:6806/17761","state":["exists","up"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":17762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":17762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":17762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":17762}]},"public_addr":"172.21.15.104:6800/17762","cluster_addr":"172.21.15.104:6801/17762","heartbeat_back_addr":"172.21.15.104:6803/17762","heartbeat_front_addr":"172.21.15.104:6802/17762","state":["exists","up"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":17845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":17845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":17845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":17845}]},"public_addr":"172.21.15.174:6808/17845","cluster_addr":"172.21.15.174:6809/17845","heartbeat_back_addr":"172.21.15.174:6811/17845","heartbeat_front_addr":"172.21.15.174:6810/17845","state":["exists","up"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":17847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6801","nonce":17847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":17847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":17847}]},"public_addr":"172.21.15.174:6800/17847","cluster_addr":"172.21.15.174:6801/17847","heartbeat_back_addr":"172.21.15.174:6803/17847","heartbeat_front_addr":"172.21.15.174:6802/17847","state":["exists","up"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":17846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":17846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":17846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":17846}]},"public_addr":"172.21.15.174:6804/17846","cluster_addr":"172.21.15.174:6805/17846","heartbeat_back_addr":"172.21.15.174:6807/17846","heartbeat_front_addr":"172.21.15.174:6806/17846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.237413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.242543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.214389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.213881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.183415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.230911+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:21:34.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 >> v2:172.21.15.174:6812/17603 conn(0x7fa76c035ab0 msgr2=0x7fa76c037f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1290198715 >> v2:172.21.15.174:6812/17603 conn(0x7fa76c035ab0 0x7fa76c037f60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa788064f90 tx=0x7fa778007ea0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 10 monclient: shutdown 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 msgr2=0x7fa788136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa788136c60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa78400ec80 tx=0x7fa7840091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 shutdown_connections 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1290198715 >> v2:172.21.15.174:6812/17603 conn(0x7fa76c035ab0 0x7fa76c037f60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa788130460 0x7fa788132ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa78812f980 0x7fa7881329a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 --2- 172.21.15.104:0/1290198715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa78809a5b0 0x7fa788136c60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 >> 172.21.15.104:0/1290198715 conn(0x7fa7880a12c0 msgr2=0x7fa78809f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:34.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 shutdown_connections 2024-09-18T14:21:34.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:34.952+0000 7fa78f44e700 1 -- 172.21.15.104:0/1290198715 wait complete. 2024-09-18T14:21:34.972 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T14:21:34.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient: get_monmap_and_config 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient: monmap: 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:35.071724+0000 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:35.071724+0000 2024-09-18T14:21:35.075 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 1 Processor -- start 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 1 -- start start 2024-09-18T14:21:35.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient: init 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0f4c0a5ca0 0x7f0f4c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c0a5ca0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c099070 0x7f0f4c09ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c099070 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f4c0a04c0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c0a64d0 con 0x7f0f4c0a04c0 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f4c099ff0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0a6070 con 0x7f0f4c099070 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f53720700 10 monclient(hunting): authenticate will time out at 816.061285s 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0f4c0a5ca0 0x7f0f4c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c099070 0x7f0f4c09ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0f4c0a5ca0 0x7f0f4c09a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50812/0 (socket says 172.21.15.104:50812) 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 1 -- 172.21.15.104:0/1968180575 learned_addr learned my addr 172.21.15.104:0/1968180575 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 10 monclient(hunting): get_auth_request con 0x7f0f4c0a5ca0 auth_method 0 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): get_auth_request con 0x7f0f4c099070 auth_method 0 2024-09-18T14:21:35.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 1 -- 172.21.15.104:0/1968180575 >> v2:172.21.15.104:3300/0 conn(0x7f0f4c0a5ca0 msgr2=0x7f0f4c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 1 --2- 172.21.15.104:0/1968180575 >> v2:172.21.15.104:3300/0 conn(0x7f0f4c0a5ca0 0x7f0f4c09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 1 -- 172.21.15.104:0/1968180575 >> v1:172.21.15.174:6789/0 conn(0x7f0f4c0a04c0 legacy=0x7f0f4c138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 1 monclient: found mon.noname-b 2024-09-18T14:21:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f50cbb700 1 -- 172.21.15.104:0/1968180575 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c09c4f0 con 0x7f0f4c099070 2024-09-18T14:21:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.068+0000 7f0f514bc700 1 --2- 172.21.15.104:0/1968180575 >> v2:172.21.15.104:3300/0 conn(0x7f0f4c0a5ca0 0x7f0f4c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f50cbb700 1 -- 172.21.15.104:0/1968180575 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c099070 msgr2=0x7f0f4c09ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f50cbb700 1 --2- 172.21.15.104:0/1968180575 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c099070 0x7f0f4c09ff80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f480098e0 tx=0x7f0f480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9009b40c9a2f557 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 1 -- 172.21.15.104:0/1968180575 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800b850 con 0x7f0f4c099070 2024-09-18T14:21:35.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:35.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: dump: 2024-09-18T14:21:35.092 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 1 -- 172.21.15.104:0/1968180575 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4800b9b0 con 0x7f0f4c099070 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 5 monclient: authenticate success, global_id 4229 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 1 -- 172.21.15.104:0/1968180575 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800bd50 con 0x7f0f4c099070 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 4 set_mon_vals no callback set 2024-09-18T14:21:35.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f3ffff700 10 monclient: dump: 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.095 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 10 monclient: shutdown 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 -- 172.21.15.104:0/1968180575 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c099070 msgr2=0x7f0f4c09ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 --2- 172.21.15.104:0/1968180575 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c099070 0x7f0f4c09ff80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f480098e0 tx=0x7f0f480091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 -- 172.21.15.104:0/1968180575 shutdown_connections 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 --2- 172.21.15.104:0/1968180575 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c099070 0x7f0f4c09ff80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 --2- 172.21.15.104:0/1968180575 >> v2:172.21.15.104:3300/0 conn(0x7f0f4c0a5ca0 0x7f0f4c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 -- 172.21.15.104:0/1968180575 >> 172.21.15.104:0/1968180575 conn(0x7f0f4c069dc0 msgr2=0x7f0f4c0a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 -- 172.21.15.104:0/1968180575 shutdown_connections 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 -- 172.21.15.104:0/1968180575 wait complete. 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 10 monclient: monmap: 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 Processor -- start 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 1 -- start start 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.072+0000 7f0f53720700 10 monclient: init 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f4c099070 0x7f0f4c141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c099070 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0a04c0 0x7f0f4c141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c0a04c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 0x7f0f4c08cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): picked mon.noname-c con 0x7f0f4c0a5ca0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0a3fb0 con 0x7f0f4c099070 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0907b0 con 0x7f0f4c0a04c0 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0916f0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient(hunting): authenticate will time out at 816.068429s 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 0x7f0f4c08cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 0x7f0f4c08cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35700/0 (socket says 172.21.15.104:35700) 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 -- 172.21.15.104:0/2098625876 learned_addr learned my addr 172.21.15.104:0/2098625876 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): get_auth_request con 0x7f0f4c0a5ca0 auth_method 0 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f4c099070 0x7f0f4c141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 10 monclient(hunting): get_auth_request con 0x7f0f4c099070 auth_method 0 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0a04c0 0x7f0f4c141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f514bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 10 monclient(hunting): get_auth_request con 0x7f0f4c0a04c0 auth_method 0 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f50cbb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 -- 172.21.15.104:0/2098625876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0a04c0 msgr2=0x7f0f4c141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0a04c0 0x7f0f4c141eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 -- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f4c099070 msgr2=0x7f0f4c141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f4c099070 0x7f0f4c141970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 monclient: found mon.noname-c 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 -- 172.21.15.104:0/2098625876 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f44009ae0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 -- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 msgr2=0x7f0f4c08cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f51cbd700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 0x7f0f4c08cec0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000c5b0 tx=0x7f0f40007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 1 -- 172.21.15.104:0/2098625876 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f400117a0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: dump: 2024-09-18T14:21:35.102 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 5 monclient: authenticate success, global_id 4232 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 1 -- 172.21.15.104:0/2098625876 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f40011900 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient: _renew_subs 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f48009420 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 1 -- 172.21.15.104:0/2098625876 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f40011cc0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3d7fa700 10 monclient: dump: 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f3f7fe700 4 set_mon_vals no callback set 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient: _renew_subs 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.076+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c08f5e0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f3d7fa700 1 -- 172.21.15.104:0/2098625876 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f0f40012630 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f3d7fa700 1 --2- 172.21.15.104:0/2098625876 >> v2:172.21.15.174:6812/17603 conn(0x7f0f30035a80 0x7f0f30037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f3d7fa700 1 -- 172.21.15.104:0/2098625876 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f0f40047b50 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f514bc700 1 --2- 172.21.15.104:0/2098625876 >> v2:172.21.15.174:6812/17603 conn(0x7f0f30035a80 0x7f0f30037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f53720700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f53720700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f53720700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4c0636d0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.080+0000 7f0f514bc700 10 monclient: get_auth_request con 0x7f0f30035a80 auth_method 0 2024-09-18T14:21:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.084+0000 7f0f514bc700 1 --2- 172.21.15.104:0/2098625876 >> v2:172.21.15.174:6812/17603 conn(0x7f0f30035a80 0x7f0f30037f30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0f440072d0 tx=0x7f0f4400b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.088+0000 7f0f3d7fa700 1 -- 172.21.15.104:0/2098625876 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0f40047e60 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.088+0000 7f0f3d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.088+0000 7f0f3d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:35.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.256+0000 7f0f53720700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:35.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.256+0000 7f0f53720700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:35.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.256+0000 7f0f53720700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.256+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f4c068c70 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.260+0000 7f0f3d7fa700 1 -- 172.21.15.104:0/2098625876 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+10108 (secure 0 0 0) 0x7f0f400108e0 con 0x7f0f4c0a5ca0 2024-09-18T14:21:35.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.260+0000 7f0f3d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:35.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.260+0000 7f0f3d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:35.267 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:35.267 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":19,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:21:34.289999+0000","last_up_change":"2024-09-18T14:21:25.123789+0000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T14:21:26.474399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T14:21:30.433492+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":17763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":17763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":17763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":17763}]},"public_addr":"172.21.15.104:6808/17763","cluster_addr":"172.21.15.104:6809/17763","heartbeat_back_addr":"172.21.15.104:6811/17763","heartbeat_front_addr":"172.21.15.104:6810/17763","state":["exists","up"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":17761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":17761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":17761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":17761}]},"public_addr":"172.21.15.104:6804/17761","cluster_addr":"172.21.15.104:6805/17761","heartbeat_back_addr":"172.21.15.104:6807/17761","heartbeat_front_addr":"172.21.15.104:6806/17761","state":["exists","up"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":17762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":17762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":17762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":17762}]},"public_addr":"172.21.15.104:6800/17762","cluster_addr":"172.21.15.104:6801/17762","heartbeat_back_addr":"172.21.15.104:6803/17762","heartbeat_front_addr":"172.21.15.104:6802/17762","state":["exists","up"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":17845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":17845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":17845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":17845}]},"public_addr":"172.21.15.174:6808/17845","cluster_addr":"172.21.15.174:6809/17845","heartbeat_back_addr":"172.21.15.174:6811/17845","heartbeat_front_addr":"172.21.15.174:6810/17845","state":["exists","up"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":17847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6801","nonce":17847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":17847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":17847}]},"public_addr":"172.21.15.174:6800/17847","cluster_addr":"172.21.15.174:6801/17847","heartbeat_back_addr":"172.21.15.174:6803/17847","heartbeat_front_addr":"172.21.15.174:6802/17847","state":["exists","up"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":17846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":17846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":17846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":17846}]},"public_addr":"172.21.15.174:6804/17846","cluster_addr":"172.21.15.174:6805/17846","heartbeat_back_addr":"172.21.15.174:6807/17846","heartbeat_front_addr":"172.21.15.174:6806/17846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.237413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.242543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.214389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.213881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.183415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.230911+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 >> v2:172.21.15.174:6812/17603 conn(0x7f0f30035a80 msgr2=0x7f0f30037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 --2- 172.21.15.104:0/2098625876 >> v2:172.21.15.174:6812/17603 conn(0x7f0f30035a80 0x7f0f30037f30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0f440072d0 tx=0x7f0f4400b040 comp rx=0 tx=0).stop 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 10 monclient: shutdown 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 msgr2=0x7f0f4c08cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 0x7f0f4c08cec0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000c5b0 tx=0x7f0f40007f60 comp rx=0 tx=0).stop 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 shutdown_connections 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 --2- 172.21.15.104:0/2098625876 >> v2:172.21.15.174:6812/17603 conn(0x7f0f30035a80 0x7f0f30037f30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0f4c0a5ca0 0x7f0f4c08cec0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.271 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f4c0a04c0 0x7f0f4c141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 --2- 172.21.15.104:0/2098625876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f4c099070 0x7f0f4c141970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 >> 172.21.15.104:0/2098625876 conn(0x7f0f4c069dc0 msgr2=0x7f0f4c09a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 shutdown_connections 2024-09-18T14:21:35.272 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.264+0000 7f0f53720700 1 -- 172.21.15.104:0/2098625876 wait complete. 2024-09-18T14:21:35.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T14:21:35.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T14:21:35.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T14:21:35.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T14:21:35.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-18T14:21:35.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-18T14:21:35.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient: get_monmap_and_config 2024-09-18T14:21:35.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient: monmap: 2024-09-18T14:21:35.395 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:35.392642+0000 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:35.392642+0000 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:35.396 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 Processor -- start 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 -- start start 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient: init 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0a18131a80 0x7f0a18131e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): picked mon.noname-c con 0x7f0a18131a80 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0a1806f1e0 0x7f0a1806f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): picked mon.noname-a con 0x7f0a1806f1e0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 0x7f0a1809deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): picked mon.noname-b con 0x7f0a1809da10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1806b3d0 con 0x7f0a1806f1e0 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1809f7f0 con 0x7f0a18131a80 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a181326b0 con 0x7f0a1809da10 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1f023700 10 monclient(hunting): authenticate will time out at 816.382983s 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 0x7f0a1809deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 0x7f0a1809deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35702/0 (socket says 172.21.15.104:35702) 2024-09-18T14:21:35.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 -- 172.21.15.104:0/1586558734 learned_addr learned my addr 172.21.15.104:0/1586558734 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): get_auth_request con 0x7f0a1809da10 auth_method 0 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e021700 1 --2- 172.21.15.104:0/1586558734 >> v2:172.21.15.174:3300/0 conn(0x7f0a18131a80 0x7f0a18131e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e021700 10 monclient(hunting): get_auth_request con 0x7f0a18131a80 auth_method 0 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e021700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e021700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e021700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 -- 172.21.15.104:0/1586558734 >> v2:172.21.15.174:3300/0 conn(0x7f0a18131a80 msgr2=0x7f0a18131e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 --2- 172.21.15.104:0/1586558734 >> v2:172.21.15.174:3300/0 conn(0x7f0a18131a80 0x7f0a18131e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 -- 172.21.15.104:0/1586558734 >> v2:172.21.15.104:3300/0 conn(0x7f0a1806f1e0 msgr2=0x7f0a1806f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 --2- 172.21.15.104:0/1586558734 >> v2:172.21.15.104:3300/0 conn(0x7f0a1806f1e0 0x7f0a1806f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 monclient: found mon.noname-b 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e822700 1 -- 172.21.15.104:0/1586558734 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1806b0f0 con 0x7f0a1809da10 2024-09-18T14:21:35.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.388+0000 7f0a1e021700 1 --2- 172.21.15.104:0/1586558734 >> v2:172.21.15.174:3300/0 conn(0x7f0a18131a80 0x7f0a18131e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:35.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1e822700 1 -- 172.21.15.104:0/1586558734 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 msgr2=0x7f0a1809deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1e822700 1 --2- 172.21.15.104:0/1586558734 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 0x7f0a1809deb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a1806a190 tx=0x7f0a10009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5955cc353a4dca33 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 1 -- 172.21.15.104:0/1586558734 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1000e070 con 0x7f0a1809da10 2024-09-18T14:21:35.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:35.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: dump: 2024-09-18T14:21:35.403 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 1 -- 172.21.15.104:0/1586558734 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1000bc60 con 0x7f0a1809da10 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 1 -- 172.21.15.104:0/1586558734 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a100092e0 con 0x7f0a1809da10 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1d01f700 10 monclient: dump: 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 5 monclient: authenticate success, global_id 4235 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 4 set_mon_vals no callback set 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 10 monclient: shutdown 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 -- 172.21.15.104:0/1586558734 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 msgr2=0x7f0a1809deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 --2- 172.21.15.104:0/1586558734 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 0x7f0a1809deb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a1806a190 tx=0x7f0a10009840 comp rx=0 tx=0).stop 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 -- 172.21.15.104:0/1586558734 shutdown_connections 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 --2- 172.21.15.104:0/1586558734 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a1809da10 0x7f0a1809deb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 --2- 172.21.15.104:0/1586558734 >> v2:172.21.15.104:3300/0 conn(0x7f0a1806f1e0 0x7f0a1806f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 --2- 172.21.15.104:0/1586558734 >> v2:172.21.15.174:3300/0 conn(0x7f0a18131a80 0x7f0a18131e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 -- 172.21.15.104:0/1586558734 >> 172.21.15.104:0/1586558734 conn(0x7f0a180676a0 msgr2=0x7f0a18067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 -- 172.21.15.104:0/1586558734 shutdown_connections 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.392+0000 7f0a1f023700 1 -- 172.21.15.104:0/1586558734 wait complete. 2024-09-18T14:21:35.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient: monmap: 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 Processor -- start 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 -- start start 2024-09-18T14:21:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient: init 2024-09-18T14:21:35.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:35.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a1806f1e0 0x7f0a18141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): picked mon.noname-b con 0x7f0a1806f1e0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:35.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a1809da10 0x7f0a18130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): picked mon.noname-a con 0x7f0a1809da10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a1809da10 0x7f0a18130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7fea19039700 10 monclient: get_monmap_and_config 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7fea19039700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7fea19039700 10 monclient: monmap: 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:35.401008+0000 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:35.401008+0000 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.409 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.410 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:35.410 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:35.410 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:35.410 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:35.410 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:35.410 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a1809da10 0x7f0a18130b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50840/0 (socket says 172.21.15.104:50840) 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a18131a80 0x7f0a1811a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 -- 172.21.15.104:0/3928522397 learned_addr learned my addr 172.21.15.104:0/3928522397 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): picked mon.noname-c con 0x7f0a18131a80 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18134c00 con 0x7f0a1809da10 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18142130 con 0x7f0a1806f1e0 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1811e1e0 con 0x7f0a18131a80 2024-09-18T14:21:35.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1f023700 10 monclient(hunting): authenticate will time out at 816.390342s 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): get_auth_request con 0x7f0a1809da10 auth_method 0 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 1 --2- 172.21.15.104:0/3928522397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a1806f1e0 0x7f0a18141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 10 monclient(hunting): get_auth_request con 0x7f0a1806f1e0 auth_method 0 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 1 --2- 172.21.15.104:0/3928522397 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a18131a80 0x7f0a1811a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 10 monclient(hunting): get_auth_request con 0x7f0a18131a80 auth_method 0 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e021700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1e822700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient(hunting): handle_auth_done global_id 4236 payload 824 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 -- 172.21.15.104:0/3928522397 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a18131a80 msgr2=0x7f0a1811a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 --2- 172.21.15.104:0/3928522397 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0a18131a80 0x7f0a1811a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 -- 172.21.15.104:0/3928522397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a1806f1e0 msgr2=0x7f0a18141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 --2- 172.21.15.104:0/3928522397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a1806f1e0 0x7f0a18141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 monclient: found mon.noname-a 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7f0a1d820700 1 -- 172.21.15.104:0/3928522397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10009ae0 con 0x7f0a1809da10 2024-09-18T14:21:35.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7fea19039700 1 Processor -- start 2024-09-18T14:21:35.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7fea19039700 1 -- start start 2024-09-18T14:21:35.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7fea19039700 10 monclient: init 2024-09-18T14:21:35.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.396+0000 7fea19039700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T14:21:35.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 0x7fea14132150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient(hunting): picked mon.noname-b con 0x7fea14131d80 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea14098010 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fea1409bc80 0x7fea14132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient(hunting): picked mon.noname-c con 0x7fea1409bc80 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea14132bd0 con 0x7fea14098010 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea140966a0 con 0x7fea1409bc80 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea14096890 con 0x7fea14131d80 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient(hunting): authenticate will time out at 816.392147s 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 0x7fea14132150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 0x7fea14132150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35724/0 (socket says 172.21.15.104:35724) 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 -- 172.21.15.104:0/4216158503 learned_addr learned my addr 172.21.15.104:0/4216158503 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): get_auth_request con 0x7fea14131d80 auth_method 0 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 -- 172.21.15.104:0/4216158503 >> v2:172.21.15.174:3300/0 conn(0x7fea1409bc80 msgr2=0x7fea14132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea1359e700 1 --2- 172.21.15.104:0/4216158503 >> v2:172.21.15.174:3300/0 conn(0x7fea1409bc80 0x7fea14132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 --2- 172.21.15.104:0/4216158503 >> v2:172.21.15.174:3300/0 conn(0x7fea1409bc80 0x7fea14132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 -- 172.21.15.104:0/4216158503 >> v1:172.21.15.174:6789/0 conn(0x7fea14098010 legacy=0x7fea1412ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 monclient: found mon.noname-b 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 -- 172.21.15.104:0/4216158503 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea14096b70 con 0x7fea14131d80 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea1359e700 10 monclient: get_auth_request con 0x7fea1409bc80 auth_method 0 2024-09-18T14:21:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea1359e700 1 --2- 172.21.15.104:0/4216158503 >> v2:172.21.15.174:3300/0 conn(0x7fea1409bc80 0x7fea14132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:21:35.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1d820700 1 -- 172.21.15.104:0/3928522397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a1809da10 msgr2=0x7f0a18130b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1d820700 1 --2- 172.21.15.104:0/3928522397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a1809da10 0x7f0a18130b00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800b810 tx=0x7f0a080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0800f720 con 0x7f0a1809da10 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: dump: 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0800f880 con 0x7f0a1809da10 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 5 monclient: authenticate success, global_id 4236 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 10 monclient: _renew_subs 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1811c160 con 0x7f0a1809da10 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1c81e700 4 set_mon_vals no callback set 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0800fc40 con 0x7f0a1809da10 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: dump: 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 10 monclient: _renew_subs 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1811dc20 con 0x7f0a1809da10 2024-09-18T14:21:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a18063450 con 0x7f0a1809da10 2024-09-18T14:21:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f0a0801a400 con 0x7f0a1809da10 2024-09-18T14:21:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 -- 172.21.15.104:0/4216158503 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 msgr2=0x7fea14132150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea12d9d700 1 --2- 172.21.15.104:0/4216158503 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 0x7fea14132150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea040099b0 tx=0x7fea04009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50a58cd406ec8a9a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.174:6812/17603 conn(0x7f09fc035a80 0x7f09fc037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f0a08047970 con 0x7f0a1809da10 2024-09-18T14:21:35.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.104:6804/17761 conn(0x7f09fc03b520 0x7f09fc03d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 --> v2:172.21.15.104:6804/17761 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09fc03dfe0 con 0x7f09fc03b520 2024-09-18T14:21:35.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a0804c2a0 con 0x7f0a1809da10 2024-09-18T14:21:35.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a0e7fc700 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-18T14:21:35.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 1 -- 172.21.15.104:0/4216158503 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0400bb50 con 0x7fea14131d80 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: dump: 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 1 -- 172.21.15.104:0/4216158503 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0400bcb0 con 0x7fea14131d80 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 1 -- 172.21.15.104:0/4216158503 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04011670 con 0x7fea14131d80 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea11d9b700 10 monclient: dump: 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1e822700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.104:6804/17761 conn(0x7f09fc03b520 0x7f09fc03d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1e021700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.174:6812/17603 conn(0x7f09fc035a80 0x7f09fc037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1e822700 10 monclient: get_auth_request con 0x7f09fc03b520 auth_method 0 2024-09-18T14:21:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7f0a1e021700 10 monclient: get_auth_request con 0x7f09fc035a80 auth_method 0 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 5 monclient: authenticate success, global_id 4241 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.400+0000 7fea19039700 4 set_mon_vals no callback set 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient: shutdown 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- 172.21.15.104:0/4216158503 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 msgr2=0x7fea14132150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 --2- 172.21.15.104:0/4216158503 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 0x7fea14132150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea040099b0 tx=0x7fea04009250 comp rx=0 tx=0).stop 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- 172.21.15.104:0/4216158503 shutdown_connections 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 --2- 172.21.15.104:0/4216158503 >> v2:172.21.15.174:3300/0 conn(0x7fea1409bc80 0x7fea14132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 --2- 172.21.15.104:0/4216158503 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14131d80 0x7fea14132150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- 172.21.15.104:0/4216158503 >> 172.21.15.104:0/4216158503 conn(0x7fea14067d70 msgr2=0x7fea140a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- 172.21.15.104:0/4216158503 shutdown_connections 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- 172.21.15.104:0/4216158503 wait complete. 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient: monmap: 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:35.432 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 Processor -- start 2024-09-18T14:21:35.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- start start 2024-09-18T14:21:35.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient: init 2024-09-18T14:21:35.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7f0a1e021700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.174:6812/17603 conn(0x7f09fc035a80 0x7f09fc037f30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a14006fd0 tx=0x7f0a14008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14098010 0x7fea14141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): picked mon.noname-c con 0x7fea14098010 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea1409bc80 0x7fea1412e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): picked mon.noname-a con 0x7fea1409bc80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea14131d80 0x7fea1412e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): picked mon.noname-b con 0x7fea14131d80 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:35.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea14095a50 con 0x7fea1409bc80 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140a0320 con 0x7fea14131d80 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1411e180 con 0x7fea14098010 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea19039700 10 monclient(hunting): authenticate will time out at 816.399429s 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea1409bc80 0x7fea1412e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14098010 0x7fea14141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea1409bc80 0x7fea1412e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50844/0 (socket says 172.21.15.104:50844) 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 1 -- 172.21.15.104:0/1611824206 learned_addr learned my addr 172.21.15.104:0/1611824206 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14098010 0x7fea14141240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35728/0 (socket says 172.21.15.104:35728) 2024-09-18T14:21:35.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 10 monclient(hunting): get_auth_request con 0x7fea1409bc80 auth_method 0 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 10 monclient(hunting): get_auth_request con 0x7fea14098010 auth_method 0 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1359e700 1 --2- 172.21.15.104:0/1611824206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea14131d80 0x7fea1412e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea1259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.404+0000 7fea12d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 -- 172.21.15.104:0/1611824206 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14098010 msgr2=0x7fea14141240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 --2- 172.21.15.104:0/1611824206 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fea14098010 0x7fea14141240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 -- 172.21.15.104:0/1611824206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea14131d80 msgr2=0x7fea1412e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 --2- 172.21.15.104:0/1611824206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea14131d80 0x7fea1412e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 monclient: found mon.noname-a 2024-09-18T14:21:35.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 -- 172.21.15.104:0/1611824206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea040094f0 con 0x7fea1409bc80 2024-09-18T14:21:35.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1359e700 10 monclient: get_auth_request con 0x7fea14131d80 auth_method 0 2024-09-18T14:21:35.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1359e700 1 --2- 172.21.15.104:0/1611824206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea14131d80 0x7fea1412e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:21:35.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7f0a1e822700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.104:6804/17761 conn(0x7f09fc03b520 0x7f09fc03d950 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== osd.1 v2:172.21.15.104:6804/17761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f09fc03dfe0 con 0x7f09fc03b520 2024-09-18T14:21:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 -- 172.21.15.104:0/1611824206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea1409bc80 msgr2=0x7fea1412e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1259c700 1 --2- 172.21.15.104:0/1611824206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea1409bc80 0x7fea1412e030 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fea0800c5d0 tx=0x7fea08007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea080117a0 con 0x7fea1409bc80 2024-09-18T14:21:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: dump: 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08011920 con 0x7fea1409bc80 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea19039700 5 monclient: authenticate success, global_id 4239 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea19039700 10 monclient: _renew_subs 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea19039700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1411cd70 con 0x7fea1409bc80 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea19039700 10 monclient: _renew_subs 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea19039700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1411d220 con 0x7fea1409bc80 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08011ce0 con 0x7fea1409bc80 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fe9faffd700 10 monclient: dump: 2024-09-18T14:21:35.791 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.408+0000 7fea1159a700 4 set_mon_vals no callback set 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fea08012630 con 0x7fea1409bc80 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fe9faffd700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6812/17603 conn(0x7fea00035a80 0x7fea00037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.792 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fea19039700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9f4000f20 con 0x7fea1409bc80 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fea08047cd0 con 0x7fea1409bc80 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fe9faffd700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6800/17847 conn(0x7fea0003b520 0x7fea0003d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 --> v2:172.21.15.174:6800/17847 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea0003dfe0 con 0x7fea0003b520 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea080021c0 con 0x7fea1409bc80 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fe9faffd700 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fea12d9d700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6812/17603 conn(0x7fea00035a80 0x7fea00037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fea12d9d700 10 monclient: get_auth_request con 0x7fea00035a80 auth_method 0 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fea1359e700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6800/17847 conn(0x7fea0003b520 0x7fea0003d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fea1359e700 10 monclient: get_auth_request con 0x7fea0003b520 auth_method 0 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient: get_monmap_and_config 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient: monmap: 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:35.417717+0000 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:35.417717+0000 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:35.794 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 1 Processor -- start 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 10 monclient: get_monmap_and_config 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 10 monclient: monmap: 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:35.418118+0000 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:35.418118+0000 2024-09-18T14:21:35.795 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 1 -- start start 2024-09-18T14:21:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient: init 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 1 Processor -- start 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f167bc93700 10 monclient: get_monmap_and_config 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f167bc93700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f167bc93700 10 monclient: monmap: 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:35.418859+0000 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:35.418859+0000 2024-09-18T14:21:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 1 -- start start 2024-09-18T14:21:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 10 monclient: init 2024-09-18T14:21:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T14:21:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0bc131d80 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 0x7fc0bc1370b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient(hunting): picked mon.noname-b con 0x7fc0bc098010 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc0bc132860 0x7fc0bc139600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient(hunting): picked mon.noname-a con 0x7fc0bc132860 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc096890 con 0x7fc0bc131d80 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0bc096b70 con 0x7fc0bc132860 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc09dad0 con 0x7fc0bc098010 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0c19f4700 10 monclient(hunting): authenticate will time out at 816.407769s 2024-09-18T14:21:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0ba7fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 0x7fc0bc1370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7f6dd06f7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6dc8097c60 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6dc809ae30 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6dc809b230 0x7f6dc81385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 10 monclient(hunting): picked mon.noname-c con 0x7f6dc809b230 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc809d1c0 con 0x7f6dc809ae30 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc8098030 con 0x7f6dc8097c60 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc809cf80 con 0x7f6dc809b230 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dd06f7700 10 monclient(hunting): authenticate will time out at 816.408094s 2024-09-18T14:21:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0ba7fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 0x7fc0bc1370b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35730/0 (socket says 172.21.15.104:35730) 2024-09-18T14:21:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/1389015295 learned_addr learned my addr 172.21.15.104:0/1389015295 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0ba7fc700 10 monclient(hunting): get_auth_request con 0x7fc0bc098010 auth_method 0 2024-09-18T14:21:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 1 Processor -- start 2024-09-18T14:21:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.412+0000 7fc0ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 1 -- start start 2024-09-18T14:21:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dce493700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6dc8097c60 0x7f6dc812fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.104:36404/0 (socket says 172.21.15.104:36404) 2024-09-18T14:21:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dce493700 1 -- 172.21.15.104:0/242273377 learned_addr learned my addr 172.21.15.104:0/242273377 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fea12d9d700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6812/17603 conn(0x7fea00035a80 0x7fea00037f30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fea040099b0 tx=0x7fea04016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0bb7fe700 1 --2- 172.21.15.104:0/1389015295 >> v2:172.21.15.104:3300/0 conn(0x7fc0bc132860 0x7fc0bc139600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-09-18T14:21:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/1389015295 >> v2:172.21.15.104:3300/0 conn(0x7fc0bc132860 msgr2=0x7fc0bc139600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient: init 2024-09-18T14:21:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 1 --2- 172.21.15.104:0/1389015295 >> v2:172.21.15.104:3300/0 conn(0x7fc0bc132860 0x7fc0bc139600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/1389015295 >> v1:172.21.15.174:6789/0 conn(0x7fc0bc131d80 legacy=0x7fc0bc132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 1 monclient: found mon.noname-b 2024-09-18T14:21:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/1389015295 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc0975a0 con 0x7fc0bc098010 2024-09-18T14:21:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0bb7fe700 10 monclient: get_auth_request con 0x7fc0bc132860 auth_method 0 2024-09-18T14:21:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0bb7fe700 1 --2- 172.21.15.104:0/1389015295 >> v2:172.21.15.104:3300/0 conn(0x7fc0bc132860 0x7fc0bc139600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:21:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16740a2bb0 0x7f167409a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient(hunting): picked mon.noname-a con 0x7f16740a2bb0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f167409d850 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f167409c380 0x7f167413d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient(hunting): picked mon.noname-b con 0x7f167409c380 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16740a2f80 con 0x7f167409d850 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16740999d0 con 0x7f16740a2bb0 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167409e5f0 con 0x7f167409c380 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167bc93700 10 monclient(hunting): authenticate will time out at 816.409278s 2024-09-18T14:21:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16740a2bb0 0x7f167409a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167922e700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f167409d850 0x7f167409bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.104:36410/0 (socket says 172.21.15.104:36410) 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f167922e700 1 -- 172.21.15.104:0/123542260 learned_addr learned my addr 172.21.15.104:0/123542260 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): get_auth_request con 0x7f16740a2bb0 auth_method 0 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 1 -- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f167409c380 msgr2=0x7f167413d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 1 --2- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f167409c380 0x7f167413d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 1 -- 172.21.15.104:0/123542260 >> v1:172.21.15.174:6789/0 conn(0x7f167409d850 legacy=0x7f167409bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 1 monclient: found mon.noname-a 2024-09-18T14:21:35.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 1 -- 172.21.15.104:0/123542260 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167409e730 con 0x7f16740a2bb0 2024-09-18T14:21:35.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 1 --2- 172.21.15.104:0/242273377 >> v2:172.21.15.174:3300/0 conn(0x7f6dc809b230 0x7f6dc81385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): get_auth_request con 0x7f6dc809b230 auth_method 0 2024-09-18T14:21:35.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dce493700 1 -- 172.21.15.104:0/242273377 >> v1:172.21.15.174:6789/0 conn(0x7f6dc8097c60 legacy=0x7f6dc812fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fea1359e700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6800/17847 conn(0x7fea0003b520 0x7fea0003d950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcdc92700 1 -- 172.21.15.104:0/242273377 >> v1:172.21.15.104:6789/0 conn(0x7f6dc809ae30 legacy=0x7f6dc8134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/1389015295 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 msgr2=0x7fc0bc1370b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0ba7fc700 1 --2- 172.21.15.104:0/1389015295 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 0x7fc0bc1370b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00099b0 tx=0x7fc0b0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b23c96778685690f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient(hunting): handle_auth_done global_id 4237 payload 274 2024-09-18T14:21:35.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 1 -- 172.21.15.104:0/242273377 >> v1:172.21.15.174:6789/0 conn(0x7f6dc8097c60 legacy=0x7f6dc812fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 1 -- 172.21.15.104:0/242273377 >> v1:172.21.15.104:6789/0 conn(0x7f6dc809ae30 legacy=0x7f6dc8134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 1 monclient: found mon.noname-c 2024-09-18T14:21:35.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:35.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f6dcec94700 1 -- 172.21.15.104:0/242273377 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8098770 con 0x7f6dc809b230 2024-09-18T14:21:35.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 1 -- 172.21.15.104:0/1389015295 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000bb50 con 0x7fc0bc098010 2024-09-18T14:21:35.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:35.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: dump: 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 1 -- 172.21.15.104:0/1389015295 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b000bcb0 con 0x7fc0bc098010 2024-09-18T14:21:35.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0c19f4700 5 monclient: authenticate success, global_id 4247 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0c19f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0c19f4700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0c19f4700 4 set_mon_vals no callback set 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 1 -- 172.21.15.104:0/1389015295 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0011670 con 0x7fc0bc098010 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7fc0b9ffb700 10 monclient: dump: 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.815 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f1679a2f700 1 -- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16740a2bb0 msgr2=0x7f167409a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1679a2f700 1 --2- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16740a2bb0 0x7f167409a3b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16700098e0 tx=0x7f16700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c86afe44b2a648 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f7082825700 10 monclient: get_monmap_and_config 2024-09-18T14:21:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f7082825700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.416+0000 7f7082825700 10 monclient: monmap: 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:35.423496+0000 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:35.423496+0000 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:35.817 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 1 -- 172.21.15.104:0/123542260 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167000ba70 con 0x7f16740a2bb0 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: dump: 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.818 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 5 monclient: authenticate success, global_id 4242 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 1 -- 172.21.15.104:0/123542260 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167000bbd0 con 0x7f16740a2bb0 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 4 set_mon_vals no callback set 2024-09-18T14:21:35.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 1 -- 172.21.15.104:0/123542260 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1670011420 con 0x7f16740a2bb0 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f1678a2d700 10 monclient: dump: 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.820 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient: shutdown 2024-09-18T14:21:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- 172.21.15.104:0/1389015295 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 msgr2=0x7fc0bc1370b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 --2- 172.21.15.104:0/1389015295 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 0x7fc0bc1370b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00099b0 tx=0x7fc0b0009250 comp rx=0 tx=0).stop 2024-09-18T14:21:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 Processor -- start 2024-09-18T14:21:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 -- start start 2024-09-18T14:21:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient: init 2024-09-18T14:21:35.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== osd.4 v2:172.21.15.174:6800/17847 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fea0003dfe0 con 0x7fea0003b520 2024-09-18T14:21:35.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f707c09c510 0x7f707c09c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): picked mon.noname-a con 0x7f707c09c510 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f707c095fa0 0x7f707c09ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): picked mon.noname-c con 0x7f707c095fa0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c098770 0x7f707c09d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): picked mon.noname-b con 0x7f707c098770 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707c097310 con 0x7f707c09c510 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707c095cf0 con 0x7f707c095fa0 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c09d8a0 con 0x7f707c098770 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7082825700 10 monclient(hunting): authenticate will time out at 816.413213s 2024-09-18T14:21:35.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- 172.21.15.104:0/1389015295 shutdown_connections 2024-09-18T14:21:35.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 --2- 172.21.15.104:0/1389015295 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc098010 0x7fc0bc1370b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 --2- 172.21.15.104:0/1389015295 >> v2:172.21.15.104:3300/0 conn(0x7fc0bc132860 0x7fc0bc139600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- 172.21.15.104:0/1389015295 >> 172.21.15.104:0/1389015295 conn(0x7fc0bc067d70 msgr2=0x7fc0bc0a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 10 monclient: shutdown 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 1 -- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16740a2bb0 msgr2=0x7f167409a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 1 --2- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16740a2bb0 0x7f167409a3b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16700098e0 tx=0x7f16700091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 1 -- 172.21.15.104:0/123542260 shutdown_connections 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 1 --2- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f167409c380 0x7f167413d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcec94700 1 -- 172.21.15.104:0/242273377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc809b230 msgr2=0x7f6dc81385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcec94700 1 --2- 172.21.15.104:0/242273377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc809b230 0x7f6dc81385b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6db80099e0 tx=0x7f6db80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c06923dcebc18917 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f707c09c510 0x7f707c09c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f707c09c510 0x7f707c09c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50874/0 (socket says 172.21.15.104:50874) 2024-09-18T14:21:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 1 -- 172.21.15.104:0/1124382598 learned_addr learned my addr 172.21.15.104:0/1124382598 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 10 monclient(hunting): get_auth_request con 0x7f707c09c510 auth_method 0 2024-09-18T14:21:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 1 --2- 172.21.15.104:0/123542260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16740a2bb0 0x7f167409a3b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 1 --2- 172.21.15.104:0/1124382598 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c098770 0x7f707c09d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- 172.21.15.104:0/1389015295 shutdown_connections 2024-09-18T14:21:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- 172.21.15.104:0/1389015295 wait complete. 2024-09-18T14:21:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): get_auth_request con 0x7f707c098770 auth_method 0 2024-09-18T14:21:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 1 --2- 172.21.15.104:0/1124382598 >> v2:172.21.15.174:3300/0 conn(0x7f707c095fa0 0x7f707c09ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 10 monclient(hunting): get_auth_request con 0x7f707c095fa0 auth_method 0 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f167bc93700 1 -- 172.21.15.104:0/123542260 >> 172.21.15.104:0/123542260 conn(0x7f1674068f60 msgr2=0x7f16740a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 1 -- 172.21.15.104:0/242273377 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800baa0 con 0x7f6dc809b230 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: dump: 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 5 monclient: authenticate success, global_id 4237 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 1 -- 172.21.15.104:0/242273377 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db800bc00 con 0x7f6dc809b230 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 1 -- 172.21.15.104:0/242273377 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8011420 con 0x7f6dc809b230 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 4 set_mon_vals no callback set 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dcd491700 10 monclient: dump: 2024-09-18T14:21:35.836 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f707b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 1 -- 172.21.15.104:0/1124382598 >> v2:172.21.15.174:3300/0 conn(0x7f707c095fa0 msgr2=0x7f707c09ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 1 --2- 172.21.15.104:0/1124382598 >> v2:172.21.15.174:3300/0 conn(0x7f707c095fa0 0x7f707c09ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 1 -- 172.21.15.104:0/1124382598 >> v2:172.21.15.104:3300/0 conn(0x7f707c09c510 msgr2=0x7f707c09c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 1 --2- 172.21.15.104:0/1124382598 >> v2:172.21.15.104:3300/0 conn(0x7f707c09c510 0x7f707c09c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 1 monclient: found mon.noname-b 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f7080dc2700 1 -- 172.21.15.104:0/1124382598 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c06f620 con 0x7f707c098770 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient: monmap: 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:35.839 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 10 monclient: shutdown 2024-09-18T14:21:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 1 -- 172.21.15.104:0/242273377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc809b230 msgr2=0x7f6dc81385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 1 --2- 172.21.15.104:0/242273377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc809b230 0x7f6dc81385b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6db80099e0 tx=0x7f6db80091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 1 -- 172.21.15.104:0/242273377 shutdown_connections 2024-09-18T14:21:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 1 --2- 172.21.15.104:0/242273377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc809b230 0x7f6dc81385b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7f6dd06f7700 1 -- 172.21.15.104:0/242273377 >> 172.21.15.104:0/242273377 conn(0x7f6dc8067d70 msgr2=0x7f6dc80a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 Processor -- start 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- start start 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient: init 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0bc098010 0x7fc0bc1412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): picked mon.noname-b con 0x7fc0bc098010 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0bc131d80 0x7fc0bc1300c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): picked mon.noname-a con 0x7fc0bc131d80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc132860 0x7fc0bc119b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): picked mon.noname-c con 0x7fc0bc132860 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0a0320 con 0x7fc0bc131d80 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc142140 con 0x7fc0bc098010 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc11b720 con 0x7fc0bc132860 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0c19f4700 10 monclient(hunting): authenticate will time out at 816.415655s 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0bc131d80 0x7fc0bc1300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0bc131d80 0x7fc0bc1300c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50882/0 (socket says 172.21.15.104:50882) 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/382733767 learned_addr learned my addr 172.21.15.104:0/382733767 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 10 monclient(hunting): get_auth_request con 0x7fc0bc131d80 auth_method 0 2024-09-18T14:21:35.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0baffd700 1 --2- 172.21.15.104:0/382733767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0bc098010 0x7fc0bc1412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0baffd700 10 monclient(hunting): get_auth_request con 0x7fc0bc098010 auth_method 0 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0baffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.420+0000 7fc0baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-09-18T14:21:35.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/382733767 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc132860 msgr2=0x7fc0bc119b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 --2- 172.21.15.104:0/382733767 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc0bc132860 0x7fc0bc119b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/382733767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0bc098010 msgr2=0x7fc0bc1412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 --2- 172.21.15.104:0/382733767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0bc098010 0x7fc0bc1412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 monclient: found mon.noname-a 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/382733767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b00094f0 con 0x7fc0bc131d80 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 -- 172.21.15.104:0/123542260 shutdown_connections 2024-09-18T14:21:35.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 -- 172.21.15.104:0/123542260 wait complete. 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7080dc2700 1 -- 172.21.15.104:0/1124382598 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c098770 msgr2=0x7f707c09d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient: monmap: 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:35.847 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:35.848 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:35.848 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7080dc2700 1 --2- 172.21.15.104:0/1124382598 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c098770 0x7f707c09d360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f707c095bd0 tx=0x7f70700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b33649f1958ab953 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 1 -- 172.21.15.104:0/1124382598 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070011840 con 0x7f707c098770 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: dump: 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.849 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 1 -- 172.21.15.104:0/1124382598 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70700119a0 con 0x7f707c098770 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 1 -- 172.21.15.104:0/1124382598 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070011d40 con 0x7f707c098770 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:35.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f707affd700 10 monclient: dump: 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 5 monclient: authenticate success, global_id 4250 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:35.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 4 set_mon_vals no callback set 2024-09-18T14:21:35.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 Processor -- start 2024-09-18T14:21:35.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 -- start start 2024-09-18T14:21:35.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient: init 2024-09-18T14:21:35.852 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 10 monclient: shutdown 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 -- 172.21.15.104:0/1124382598 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c098770 msgr2=0x7f707c09d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 --2- 172.21.15.104:0/1124382598 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c098770 0x7f707c09d360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f707c095bd0 tx=0x7f70700091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 -- 172.21.15.104:0/1124382598 shutdown_connections 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 --2- 172.21.15.104:0/1124382598 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c098770 0x7f707c09d360 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 --2- 172.21.15.104:0/1124382598 >> v2:172.21.15.174:3300/0 conn(0x7f707c095fa0 0x7f707c09ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 --2- 172.21.15.104:0/1124382598 >> v2:172.21.15.104:3300/0 conn(0x7f707c09c510 0x7f707c09c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 -- 172.21.15.104:0/1124382598 >> 172.21.15.104:0/1124382598 conn(0x7f707c0a1290 msgr2=0x7f707c0a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f167409c380 0x7f167411a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): picked mon.noname-b con 0x7f167409c380 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f167409d850 0x7f167411ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): picked mon.noname-a con 0x7f167409d850 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 -- 172.21.15.104:0/382733767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0bc131d80 msgr2=0x7fc0bc1300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0ba7fc700 1 --2- 172.21.15.104:0/382733767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0bc131d80 0x7fc0bc1300c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc09dc10 tx=0x7fc0b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f16740a2bb0 0x7f167411b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): picked mon.noname-c con 0x7f16740a2bb0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1674126af0 con 0x7f167409d850 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16740a0570 con 0x7f167409c380 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1674125ac0 con 0x7f16740a2bb0 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167bc93700 10 monclient(hunting): authenticate will time out at 816.418408s 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f16740a2bb0 0x7f167411b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f167409d850 0x7f167411ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f16740a2bb0 0x7f167411b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35758/0 (socket says 172.21.15.104:35758) 2024-09-18T14:21:35.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f167409d850 0x7f167411ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50888/0 (socket says 172.21.15.104:50888) 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 -- 172.21.15.104:0/846056929 learned_addr learned my addr 172.21.15.104:0/846056929 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): get_auth_request con 0x7f16740a2bb0 auth_method 0 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 10 monclient(hunting): get_auth_request con 0x7f167409d850 auth_method 0 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 1 --2- 172.21.15.104:0/846056929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f167409c380 0x7f167411a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 10 monclient(hunting): get_auth_request con 0x7f167409c380 auth_method 0 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167922e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 -- 172.21.15.104:0/1124382598 shutdown_connections 2024-09-18T14:21:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 1 -- 172.21.15.104:0/1124382598 wait complete. 2024-09-18T14:21:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0011d30 con 0x7fc0bc131d80 2024-09-18T14:21:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: dump: 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b001b470 con 0x7fc0bc131d80 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0c19f4700 5 monclient: authenticate success, global_id 4251 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0c19f4700 10 monclient: _renew_subs 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0c19f4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc11d1e0 con 0x7fc0bc131d80 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b001b830 con 0x7fc0bc131d80 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0a2ffd700 10 monclient: dump: 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0c19f4700 10 monclient: _renew_subs 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0c19f4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc11c2f0 con 0x7fc0bc131d80 2024-09-18T14:21:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f1679a2f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 -- 172.21.15.104:0/846056929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f167409c380 msgr2=0x7f167411a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 --2- 172.21.15.104:0/846056929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f167409c380 0x7f167411a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 -- 172.21.15.104:0/846056929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f167409d850 msgr2=0x7f167411ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 --2- 172.21.15.104:0/846056929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f167409d850 0x7f167411ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 monclient: found mon.noname-c 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f167a230700 1 -- 172.21.15.104:0/846056929 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1670009420 con 0x7f16740a2bb0 2024-09-18T14:21:35.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7fc0b97fa700 4 set_mon_vals no callback set 2024-09-18T14:21:35.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.424+0000 7f7082825700 10 monclient: monmap: 2024-09-18T14:21:35.864 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.864 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0c19f4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc09c000f20 con 0x7fc0bc131d80 2024-09-18T14:21:35.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 Processor -- start 2024-09-18T14:21:35.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 -- 172.21.15.104:0/242273377 shutdown_connections 2024-09-18T14:21:35.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 -- start start 2024-09-18T14:21:35.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient: init 2024-09-18T14:21:35.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 -- 172.21.15.104:0/242273377 wait complete. 2024-09-18T14:21:35.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:35.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c095fa0 0x7f707c12fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): picked mon.noname-c con 0x7f707c095fa0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f707c098770 0x7f707c11a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): picked mon.noname-a con 0x7f707c098770 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f707c09c510 0x7f707c11aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): picked mon.noname-b con 0x7f707c09c510 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c11de30 con 0x7f707c098770 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c11c610 con 0x7f707c09c510 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c13dba0 con 0x7f707c095fa0 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7082825700 10 monclient(hunting): authenticate will time out at 816.420949s 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient: build_initial_monmap 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient: monmap: 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:35.869 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:35.870 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f707c09c510 0x7f707c11aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fc0b001bc20 con 0x7fc0bc131d80 2024-09-18T14:21:35.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 --> v2:172.21.15.104:6804/17761 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0a18063c80 con 0x7f09fc03b520 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 Processor -- start 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f707c09c510 0x7f707c11aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.104:38700/0 (socket says 172.21.15.104:38700) 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 -- 172.21.15.104:0/2302180441 learned_addr learned my addr 172.21.15.104:0/2302180441 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): get_auth_request con 0x7f707c09c510 auth_method 0 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167a230700 1 -- 172.21.15.104:0/846056929 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f16740a2bb0 msgr2=0x7f167411b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167a230700 1 --2- 172.21.15.104:0/846056929 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f16740a2bb0 0x7f167411b060 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f166c00bc30 tx=0x7f166c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 1 --2- 172.21.15.104:0/2302180441 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c095fa0 0x7f707c12fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 10 monclient(hunting): get_auth_request con 0x7f707c095fa0 auth_method 0 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0a2ffd700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.174:6812/17603 conn(0x7fc0a8035ad0 0x7fc0a8037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fc0b0049f20 con 0x7fc0bc131d80 2024-09-18T14:21:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c0092e0 con 0x7f16740a2bb0 2024-09-18T14:21:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a0e7fc700 1 -- 172.21.15.104:0/3928522397 <== osd.1 v2:172.21.15.104:6804/17761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0a18063c80 con 0x7f09fc03b520 2024-09-18T14:21:35.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 1 --2- 172.21.15.104:0/2302180441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f707c098770 0x7f707c11a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 10 monclient(hunting): get_auth_request con 0x7f707c098770 auth_method 0 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f707b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 -- 172.21.15.104:0/2302180441 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c095fa0 msgr2=0x7f707c12fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 --2- 172.21.15.104:0/2302180441 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f707c095fa0 0x7f707c12fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 -- 172.21.15.104:0/2302180441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f707c098770 msgr2=0x7f707c11a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 --2- 172.21.15.104:0/2302180441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f707c098770 0x7f707c11a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 monclient: found mon.noname-b 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:35.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f7080dc2700 1 -- 172.21.15.104:0/2302180441 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070009420 con 0x7f707c09c510 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: dump: 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.877 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166c00c840 con 0x7f16740a2bb0 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c00cc20 con 0x7f16740a2bb0 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f1669ffb700 10 monclient: dump: 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.878 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f166b7fe700 4 set_mon_vals no callback set 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 >> v2:172.21.15.104:6804/17761 conn(0x7f09fc03b520 msgr2=0x7f09fc03d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.104:6804/17761 conn(0x7f09fc03b520 0x7f09fc03d950 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 >> v2:172.21.15.174:6812/17603 conn(0x7f09fc035a80 msgr2=0x7f09fc037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 --2- 172.21.15.104:0/3928522397 >> v2:172.21.15.174:6812/17603 conn(0x7f09fc035a80 0x7f09fc037f30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a14006fd0 tx=0x7f0a14008040 comp rx=0 tx=0).stop 2024-09-18T14:21:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 10 monclient: shutdown 2024-09-18T14:21:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a1809da10 msgr2=0x7f0a18130b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 --2- 172.21.15.104:0/3928522397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a1809da10 0x7f0a18130b00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800b810 tx=0x7f0a080091d0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 -- start start 2024-09-18T14:21:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient: init 2024-09-18T14:21:35.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc8097c60 0x7f6dc812ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): picked mon.noname-b con 0x7f6dc8097c60 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6dc809ae30 0x7f6dc813f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): picked mon.noname-c con 0x7f6dc809ae30 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6dc809b230 0x7f6dc8119a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): picked mon.noname-a con 0x7f6dc809b230 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc811af10 con 0x7f6dc809b230 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc811d660 con 0x7f6dc8097c60 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:35.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc811d0a0 con 0x7f6dc809ae30 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dd06f7700 10 monclient(hunting): authenticate will time out at 816.422845s 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6dc809ae30 0x7f6dc813f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6dc809b230 0x7f6dc8119a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6dc809ae30 0x7f6dc813f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35782/0 (socket says 172.21.15.104:35782) 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 1 -- 172.21.15.104:0/2763630763 learned_addr learned my addr 172.21.15.104:0/2763630763 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 10 monclient(hunting): get_auth_request con 0x7f6dc809ae30 auth_method 0 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dce493700 1 --2- 172.21.15.104:0/2763630763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc8097c60 0x7f6dc812ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): get_auth_request con 0x7f6dc809b230 auth_method 0 2024-09-18T14:21:35.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dce493700 10 monclient(hunting): get_auth_request con 0x7f6dc8097c60 auth_method 0 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dce493700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dce493700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dce493700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcdc92700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:35.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1e822700 1 -- 172.21.15.104:0/3928522397 reap_dead start 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 shutdown_connections 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 >> 172.21.15.104:0/3928522397 conn(0x7f0a180676a0 msgr2=0x7f0a1806e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0a2ffd700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.104:6800/17762 conn(0x7fc0a803b670 0x7fc0a803daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 --> v2:172.21.15.104:6800/17762 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0a803e130 con 0x7fc0a803b670 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0b004a1c0 con 0x7fc0bc131d80 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0a2ffd700 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0baffd700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.174:6812/17603 conn(0x7fc0a8035ad0 0x7fc0a8037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0baffd700 10 monclient: get_auth_request con 0x7fc0a8035ad0 auth_method 0 2024-09-18T14:21:35.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0bb7fe700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.104:6800/17762 conn(0x7fc0a803b670 0x7fc0a803daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7fc0bb7fe700 10 monclient: get_auth_request con 0x7fc0a803b670 auth_method 0 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167bc93700 5 monclient: authenticate success, global_id 4253 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167bc93700 10 monclient: _renew_subs 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167bc93700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167411c100 con 0x7f16740a2bb0 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167bc93700 10 monclient: _renew_subs 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167bc93700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167413d8a0 con 0x7f16740a2bb0 2024-09-18T14:21:35.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-18T14:21:35.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 1 -- 172.21.15.104:0/2763630763 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6dc809ae30 msgr2=0x7f6dc813f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 1 --2- 172.21.15.104:0/2763630763 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6dc809ae30 0x7f6dc813f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 1 -- 172.21.15.104:0/2763630763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc8097c60 msgr2=0x7f6dc812ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 1 --2- 172.21.15.104:0/2763630763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6dc8097c60 0x7f6dc812ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient: _finish_hunting 0 2024-09-18T14:21:35.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 1 monclient: found mon.noname-a 2024-09-18T14:21:35.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:35.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f6dcec94700 1 -- 172.21.15.104:0/2763630763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8009420 con 0x7f6dc809b230 2024-09-18T14:21:35.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 shutdown_connections 2024-09-18T14:21:35.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.428+0000 7f0a1f023700 1 -- 172.21.15.104:0/3928522397 wait complete. 2024-09-18T14:21:35.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f167bc93700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:35.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16740636d0 con 0x7f16740a2bb0 2024-09-18T14:21:35.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f166c012070 con 0x7f16740a2bb0 2024-09-18T14:21:35.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7080dc2700 1 -- 172.21.15.104:0/2302180441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f707c09c510 msgr2=0x7f707c11aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.888 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7080dc2700 1 --2- 172.21.15.104:0/2302180441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f707c09c510 0x7f707c11aa40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f707c09d9e0 tx=0x7f7070017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1669ffb700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6812/17603 conn(0x7f1658035ad0 0x7f1658037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f166c047ea0 con 0x7f16740a2bb0 2024-09-18T14:21:35.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1669ffb700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6804/17846 conn(0x7f165803b6c0 0x7f165803daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 --> v2:172.21.15.174:6804/17846 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f165803e180 con 0x7f165803b6c0 2024-09-18T14:21:35.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f166c04a430 con 0x7f16740a2bb0 2024-09-18T14:21:35.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1669ffb700 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-18T14:21:35.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7fc0baffd700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.174:6812/17603 conn(0x7fc0a8035ad0 0x7fc0a8037f80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4007fd0 tx=0x7fc0b40026a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707001a450 con 0x7f707c09c510 2024-09-18T14:21:35.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:35.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1679a2f700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6812/17603 conn(0x7f1658035ad0 0x7f1658037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: dump: 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 5 monclient: authenticate success, global_id 4252 2024-09-18T14:21:35.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 10 monclient: _renew_subs 2024-09-18T14:21:35.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:35.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c096660 con 0x7f707c09c510 2024-09-18T14:21:35.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f1679a2f700 10 monclient: get_auth_request con 0x7f1658035ad0 auth_method 0 2024-09-18T14:21:35.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 10 monclient: _renew_subs 2024-09-18T14:21:35.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:35.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c11cc60 con 0x7f707c09c510 2024-09-18T14:21:35.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f167922e700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6804/17846 conn(0x7f165803b6c0 0x7f165803daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707001a5b0 con 0x7f707c09c510 2024-09-18T14:21:35.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707001a950 con 0x7f707c09c510 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f707a7fc700 4 set_mon_vals no callback set 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: dump: 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.894 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.895 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f167922e700 10 monclient: get_auth_request con 0x7f165803b6c0 auth_method 0 2024-09-18T14:21:35.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dcec94700 1 -- 172.21.15.104:0/2763630763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6dc809b230 msgr2=0x7f6dc8119a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:35.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dcec94700 1 --2- 172.21.15.104:0/2763630763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6dc809b230 0x7f6dc8119a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6dc809d5a0 tx=0x7f6db8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:35.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f705c000f20 con 0x7f707c09c510 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8011e60 con 0x7f6dc809b230 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: dump: 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.897 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 5 monclient: authenticate success, global_id 4260 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db801c880 con 0x7f6dc809b230 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 10 monclient: _renew_subs 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db801cc00 con 0x7f6dc809b230 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc811d380 con 0x7f6dc809b230 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dccc90700 4 set_mon_vals no callback set 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 10 monclient: _renew_subs 2024-09-18T14:21:35.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc812f4b0 con 0x7f6dc809b230 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 10 monclient: dump: 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:35.899 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:35.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f707001acc0 con 0x7f707c09c510 2024-09-18T14:21:35.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6812/17603 conn(0x7f7068035ab0 0x7f7068037f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f70700494c0 con 0x7f707c09c510 2024-09-18T14:21:35.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 0x7f706803d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 --> v2:172.21.15.174:6808/17845 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f706803e010 con 0x7f706803b550 2024-09-18T14:21:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7fc0bb7fe700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.104:6800/17762 conn(0x7fc0a803b670 0x7fc0a803daa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f707002d3f0 con 0x7f707c09c510 2024-09-18T14:21:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f7063fff700 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-18T14:21:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f6db801a450 con 0x7f6dc809b230 2024-09-18T14:21:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f707b7fe700 1 -- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 msgr2=0x7f706803d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:21:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f707b7fe700 1 -- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 msgr2=0x7f706803d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:21:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f707b7fe700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 0x7f706803d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f707b7fe700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 0x7f706803d980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.174:6812/17603 conn(0x7f6dac035ad0 0x7f6dac037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f6db804a4b0 con 0x7f6dc809b230 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.432+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6dc8064b30 con 0x7f6dc809b230 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dbe7fc700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.104:6808/17763 conn(0x7f6dac03b5f0 0x7f6dac03da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 --> v2:172.21.15.104:6808/17763 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dac03e0b0 con 0x7f6dac03b5f0 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f707bfff700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6812/17603 conn(0x7f7068035ab0 0x7f7068037f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f707bfff700 10 monclient: get_auth_request con 0x7f7068035ab0 auth_method 0 2024-09-18T14:21:35.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6db801ae60 con 0x7f6dc809b230 2024-09-18T14:21:35.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dbe7fc700 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-18T14:21:35.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f1679a2f700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6812/17603 conn(0x7f1658035ad0 0x7f1658037f80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16700030c0 tx=0x7f1670016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dce493700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.174:6812/17603 conn(0x7f6dac035ad0 0x7f6dac037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dce493700 10 monclient: get_auth_request con 0x7f6dac035ad0 auth_method 0 2024-09-18T14:21:35.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== osd.2 v2:172.21.15.104:6800/17762 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fc0a803e130 con 0x7fc0a803b670 2024-09-18T14:21:35.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dcdc92700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.104:6808/17763 conn(0x7f6dac03b5f0 0x7f6dac03da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dcdc92700 10 monclient: get_auth_request con 0x7f6dac03b5f0 auth_method 0 2024-09-18T14:21:35.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f167922e700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6804/17846 conn(0x7f165803b6c0 0x7f165803daf0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f707bfff700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6812/17603 conn(0x7f7068035ab0 0x7f7068037f60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f706c006d40 tx=0x7f706c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.436+0000 7f6dce493700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.174:6812/17603 conn(0x7f6dac035ad0 0x7f6dac037f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4001a00 tx=0x7f6dc4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.440+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== osd.5 v2:172.21.15.174:6804/17846 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f165803e180 con 0x7f165803b6c0 2024-09-18T14:21:35.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.440+0000 7f6dcdc92700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.104:6808/17763 conn(0x7f6dac03b5f0 0x7f6dac03da20 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.440+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== osd.0 v2:172.21.15.104:6808/17763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f6dac03e0b0 con 0x7f6dac03b5f0 2024-09-18T14:21:35.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.440+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 --> v2:172.21.15.174:6800/17847 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9f4002ba0 con 0x7fea0003b520 2024-09-18T14:21:35.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fe9faffd700 1 -- 172.21.15.104:0/1611824206 <== osd.4 v2:172.21.15.174:6800/17847 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe9f4002ba0 con 0x7fea0003b520 2024-09-18T14:21:35.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6800/17847 conn(0x7fea0003b520 msgr2=0x7fea0003d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6800/17847 conn(0x7fea0003b520 0x7fea0003d950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6812/17603 conn(0x7fea00035a80 msgr2=0x7fea00037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 --2- 172.21.15.104:0/1611824206 >> v2:172.21.15.174:6812/17603 conn(0x7fea00035a80 0x7fea00037f30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fea040099b0 tx=0x7fea04016000 comp rx=0 tx=0).stop 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 10 monclient: shutdown 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea1409bc80 msgr2=0x7fea1412e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 --2- 172.21.15.104:0/1611824206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea1409bc80 0x7fea1412e030 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fea0800c5d0 tx=0x7fea08007f60 comp rx=0 tx=0).stop 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea1359e700 1 -- 172.21.15.104:0/1611824206 reap_dead start 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 shutdown_connections 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 >> 172.21.15.104:0/1611824206 conn(0x7fea14067d70 msgr2=0x7fea1409f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 shutdown_connections 2024-09-18T14:21:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.444+0000 7fea19039700 1 -- 172.21.15.104:0/1611824206 wait complete. 2024-09-18T14:21:35.910 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:35.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:21:35.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 --> v2:172.21.15.104:6800/17762 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc09c002bd0 con 0x7fc0a803b670 2024-09-18T14:21:35.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0a2ffd700 1 -- 172.21.15.104:0/382733767 <== osd.2 v2:172.21.15.104:6800/17762 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc09c002bd0 con 0x7fc0a803b670 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 >> v2:172.21.15.104:6800/17762 conn(0x7fc0a803b670 msgr2=0x7fc0a803daa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.104:6800/17762 conn(0x7fc0a803b670 0x7fc0a803daa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 >> v2:172.21.15.174:6812/17603 conn(0x7fc0a8035ad0 msgr2=0x7fc0a8037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 --2- 172.21.15.104:0/382733767 >> v2:172.21.15.174:6812/17603 conn(0x7fc0a8035ad0 0x7fc0a8037f80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4007fd0 tx=0x7fc0b40026a0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 10 monclient: shutdown 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0bc131d80 msgr2=0x7fc0bc1300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 --2- 172.21.15.104:0/382733767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0bc131d80 0x7fc0bc1300c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc09dc10 tx=0x7fc0b0016000 comp rx=0 tx=0).stop 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0bb7fe700 1 -- 172.21.15.104:0/382733767 reap_dead start 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 shutdown_connections 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 >> 172.21.15.104:0/382733767 conn(0x7fc0bc067d70 msgr2=0x7fc0bc134da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 shutdown_connections 2024-09-18T14:21:35.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.448+0000 7fc0c19f4700 1 -- 172.21.15.104:0/382733767 wait complete. 2024-09-18T14:21:35.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.452+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 --> v2:172.21.15.104:6808/17763 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6dc8098a10 con 0x7f6dac03b5f0 2024-09-18T14:21:35.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.452+0000 7f6dbe7fc700 1 -- 172.21.15.104:0/2763630763 <== osd.0 v2:172.21.15.104:6808/17763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6dc8098a10 con 0x7f6dac03b5f0 2024-09-18T14:21:35.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 >> v2:172.21.15.104:6808/17763 conn(0x7f6dac03b5f0 msgr2=0x7f6dac03da20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.104:6808/17763 conn(0x7f6dac03b5f0 0x7f6dac03da20 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 >> v2:172.21.15.174:6812/17603 conn(0x7f6dac035ad0 msgr2=0x7f6dac037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 --2- 172.21.15.104:0/2763630763 >> v2:172.21.15.174:6812/17603 conn(0x7f6dac035ad0 0x7f6dac037f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4001a00 tx=0x7f6dc4009040 comp rx=0 tx=0).stop 2024-09-18T14:21:35.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 10 monclient: shutdown 2024-09-18T14:21:35.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6dc809b230 msgr2=0x7f6dc8119a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 --2- 172.21.15.104:0/2763630763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6dc809b230 0x7f6dc8119a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6dc809d5a0 tx=0x7f6db8016000 comp rx=0 tx=0).stop 2024-09-18T14:21:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dcec94700 1 -- 172.21.15.104:0/2763630763 reap_dead start 2024-09-18T14:21:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 shutdown_connections 2024-09-18T14:21:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 >> 172.21.15.104:0/2763630763 conn(0x7f6dc8067d70 msgr2=0x7f6dc809dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 shutdown_connections 2024-09-18T14:21:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.456+0000 7f6dd06f7700 1 -- 172.21.15.104:0/2763630763 wait complete. 2024-09-18T14:21:35.915 INFO:teuthology.orchestra.run.smithi104.stdout:47244640259 2024-09-18T14:21:35.916 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T14:21:35.916 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:35.917 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:21:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 --> v2:172.21.15.174:6804/17846 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f167411b5a0 con 0x7f165803b6c0 2024-09-18T14:21:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f1669ffb700 1 -- 172.21.15.104:0/846056929 <== osd.5 v2:172.21.15.174:6804/17846 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f167411b5a0 con 0x7f165803b6c0 2024-09-18T14:21:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6804/17846 conn(0x7f165803b6c0 msgr2=0x7f165803daf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6804/17846 conn(0x7f165803b6c0 0x7f165803daf0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6812/17603 conn(0x7f1658035ad0 msgr2=0x7f1658037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 --2- 172.21.15.104:0/846056929 >> v2:172.21.15.174:6812/17603 conn(0x7f1658035ad0 0x7f1658037f80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16700030c0 tx=0x7f1670016000 comp rx=0 tx=0).stop 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 10 monclient: shutdown 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f16740a2bb0 msgr2=0x7f167411b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 --2- 172.21.15.104:0/846056929 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f16740a2bb0 0x7f167411b060 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f166c00bc30 tx=0x7f166c009840 comp rx=0 tx=0).stop 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167a230700 1 -- 172.21.15.104:0/846056929 reap_dead start 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 shutdown_connections 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 >> 172.21.15.104:0/846056929 conn(0x7f1674068f60 msgr2=0x7f167409ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 shutdown_connections 2024-09-18T14:21:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.460+0000 7f167bc93700 1 -- 172.21.15.104:0/846056929 wait complete. 2024-09-18T14:21:35.920 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:35.920 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:21:35.921 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:35.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T14:21:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.636+0000 7f707b7fe700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 0x7f706803d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.636+0000 7f707b7fe700 10 monclient: get_auth_request con 0x7f706803b550 auth_method 0 2024-09-18T14:21:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.640+0000 7f707b7fe700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 0x7f706803d980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.640+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== osd.3 v2:172.21.15.174:6808/17845 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f706803e010 con 0x7f706803b550 2024-09-18T14:21:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 --> v2:172.21.15.174:6808/17845 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f705c002bd0 con 0x7f706803b550 2024-09-18T14:21:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7063fff700 1 -- 172.21.15.104:0/2302180441 <== osd.3 v2:172.21.15.174:6808/17845 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f705c002bd0 con 0x7f706803b550 2024-09-18T14:21:35.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 msgr2=0x7f706803d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6808/17845 conn(0x7f706803b550 0x7f706803d980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:35.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6812/17603 conn(0x7f7068035ab0 msgr2=0x7f7068037f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 --2- 172.21.15.104:0/2302180441 >> v2:172.21.15.174:6812/17603 conn(0x7f7068035ab0 0x7f7068037f60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f706c006d40 tx=0x7f706c008040 comp rx=0 tx=0).stop 2024-09-18T14:21:35.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 10 monclient: shutdown 2024-09-18T14:21:35.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f707c09c510 msgr2=0x7f707c11aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 --2- 172.21.15.104:0/2302180441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f707c09c510 0x7f707c11aa40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f707c09d9e0 tx=0x7f7070017000 comp rx=0 tx=0).stop 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7080dc2700 1 -- 172.21.15.104:0/2302180441 reap_dead start 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 shutdown_connections 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 >> 172.21.15.104:0/2302180441 conn(0x7f707c0a1290 msgr2=0x7f707c09fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 shutdown_connections 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:35.664+0000 7f7082825700 1 -- 172.21.15.104:0/2302180441 wait complete. 2024-09-18T14:21:35.925 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:35.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:21:36.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.084+0000 7febebef9700 10 monclient: get_monmap_and_config 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.084+0000 7febebef9700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.084+0000 7febebef9700 10 monclient: monmap: 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:36.089847+0000 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:36.089847+0000 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:36.094 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:36.094 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:36.094 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:36.094 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.084+0000 7febebef9700 1 Processor -- start 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.084+0000 7febebef9700 1 -- start start 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.084+0000 7febebef9700 10 monclient: init 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.084+0000 7febebef9700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 0x7febe409a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient(hunting): picked mon.noname-b con 0x7febe40a2c30 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febe406f240 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7febe406f610 0x7febe4067db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient(hunting): picked mon.noname-c con 0x7febe406f610 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe409ea50 con 0x7febe406f240 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe409c7a0 con 0x7febe406f610 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe4099cb0 con 0x7febe40a2c30 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febebef9700 10 monclient(hunting): authenticate will time out at 817.080177s 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 0x7febe409a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 0x7febe409a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35786/0 (socket says 172.21.15.104:35786) 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 -- 172.21.15.104:0/2452639704 learned_addr learned my addr 172.21.15.104:0/2452639704 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.096 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): get_auth_request con 0x7febe40a2c30 auth_method 0 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 1 --2- 172.21.15.104:0/2452639704 >> v2:172.21.15.174:3300/0 conn(0x7febe406f610 0x7febe4067db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 10 monclient(hunting): get_auth_request con 0x7febe406f610 auth_method 0 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febea496700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 -- 172.21.15.104:0/2452639704 >> v2:172.21.15.174:3300/0 conn(0x7febe406f610 msgr2=0x7febe4067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 --2- 172.21.15.104:0/2452639704 >> v2:172.21.15.174:3300/0 conn(0x7febe406f610 0x7febe4067db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 -- 172.21.15.104:0/2452639704 >> v1:172.21.15.104:6789/0 conn(0x7febe406f240 legacy=0x7febe40676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 monclient: found mon.noname-b 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 -- 172.21.15.104:0/2452639704 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40a3000 con 0x7febe40a2c30 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient: get_monmap_and_config 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient: monmap: 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:36.092787+0000 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:36.092787+0000 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient: get_monmap_and_config 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient: monmap: 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:36.093233+0000 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:36.093233+0000 2024-09-18T14:21:36.100 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 Processor -- start 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 -- start start 2024-09-18T14:21:36.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient: init 2024-09-18T14:21:36.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 1 Processor -- start 2024-09-18T14:21:36.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 1 -- start start 2024-09-18T14:21:36.102 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient: init 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7e7009c510 0x7f7e7009c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): picked mon.noname-c con 0x7f7e7009c510 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 0x7f7e7009ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): picked mon.noname-b con 0x7f7e70095fa0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7e70098770 0x7f7e7009d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): picked mon.noname-a con 0x7f7e70098770 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700973a0 con 0x7f7e70098770 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e70095cf0 con 0x7f7e7009c510 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7009d8a0 con 0x7f7e70095fa0 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e756ac700 10 monclient(hunting): authenticate will time out at 817.082792s 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf7c0a1900 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf7c09ad20 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcf7c137e40 0x7fcf7c13a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient(hunting): picked mon.noname-a con 0x7fcf7c137e40 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7e7009c510 0x7f7e7009c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7e70098770 0x7f7e7009d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7e7009c510 0x7f7e7009c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.104:38728/0 (socket says 172.21.15.104:38728) 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 -- 172.21.15.104:0/363051274 learned_addr learned my addr 172.21.15.104:0/363051274 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): get_auth_request con 0x7f7e7009c510 auth_method 0 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 1 --2- 172.21.15.104:0/363051274 >> v2:172.21.15.104:3300/0 conn(0x7f7e70098770 0x7f7e7009d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50926/0 (socket says 172.21.15.104:50926) 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c132610 con 0x7fcf7c0a1900 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c099510 con 0x7fcf7c09ad20 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7c0a1cd0 con 0x7fcf7c137e40 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf8298a700 10 monclient(hunting): authenticate will time out at 817.083054s 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf81988700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcf7c0a1900 0x7fcf7c131f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47444/0 (socket says 172.21.15.104:47444) 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf81988700 1 -- 172.21.15.104:0/3125379452 learned_addr learned my addr 172.21.15.104:0/3125379452 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 1 --2- 172.21.15.104:0/3125379452 >> v2:172.21.15.104:3300/0 conn(0x7fcf7c137e40 0x7fcf7c13a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): get_auth_request con 0x7fcf7c137e40 auth_method 0 2024-09-18T14:21:36.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 10 monclient(hunting): get_auth_request con 0x7f7e70098770 auth_method 0 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 1 --2- 172.21.15.104:0/363051274 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 0x7f7e7009ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 10 monclient(hunting): get_auth_request con 0x7f7e70095fa0 auth_method 0 2024-09-18T14:21:36.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 -- 172.21.15.104:0/2452639704 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 msgr2=0x7febe409a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7febe9c95700 1 --2- 172.21.15.104:0/2452639704 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 0x7febe409a980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7febdc009f20 tx=0x7febdc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28c85ca95155f0fb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 -- 172.21.15.104:0/363051274 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 msgr2=0x7f7e7009ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 --2- 172.21.15.104:0/363051274 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 0x7f7e7009ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 -- 172.21.15.104:0/363051274 >> v2:172.21.15.104:3300/0 conn(0x7f7e70098770 msgr2=0x7f7e7009d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 --2- 172.21.15.104:0/363051274 >> v2:172.21.15.104:3300/0 conn(0x7f7e70098770 0x7f7e7009d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 monclient: found mon.noname-c 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7f7e6effd700 1 -- 172.21.15.104:0/363051274 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7006f620 con 0x7f7e7009c510 2024-09-18T14:21:36.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 1 -- 172.21.15.104:0/3125379452 >> v1:172.21.15.174:6789/0 conn(0x7fcf7c09ad20 legacy=0x7fcf7c09b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 1 -- 172.21.15.104:0/3125379452 >> v1:172.21.15.104:6789/0 conn(0x7fcf7c0a1900 legacy=0x7fcf7c131f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 1 monclient: found mon.noname-a 2024-09-18T14:21:36.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.088+0000 7fcf82189700 1 -- 172.21.15.104:0/3125379452 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c09da10 con 0x7fcf7c137e40 2024-09-18T14:21:36.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 1 -- 172.21.15.104:0/2452639704 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc00e070 con 0x7febe40a2c30 2024-09-18T14:21:36.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: dump: 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 5 monclient: authenticate success, global_id 4262 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 1 -- 172.21.15.104:0/2452639704 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc00bbe0 con 0x7febe40a2c30 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 4 set_mon_vals no callback set 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 1 -- 172.21.15.104:0/2452639704 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc0092e0 con 0x7febe40a2c30 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe8c93700 10 monclient: dump: 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.114 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient: shutdown 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- 172.21.15.104:0/2452639704 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 msgr2=0x7febe409a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 --2- 172.21.15.104:0/2452639704 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 0x7febe409a980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7febdc009f20 tx=0x7febdc009840 comp rx=0 tx=0).stop 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- 172.21.15.104:0/2452639704 shutdown_connections 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 --2- 172.21.15.104:0/2452639704 >> v2:172.21.15.174:3300/0 conn(0x7febe406f610 0x7febe4067db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 --2- 172.21.15.104:0/2452639704 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe40a2c30 0x7febe409a980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- 172.21.15.104:0/2452639704 >> 172.21.15.104:0/2452639704 conn(0x7febe4068f60 msgr2=0x7febe40a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- 172.21.15.104:0/2452639704 shutdown_connections 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- 172.21.15.104:0/2452639704 wait complete. 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient: monmap: 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:36.116 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf82189700 1 -- 172.21.15.104:0/3125379452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c137e40 msgr2=0x7fcf7c13a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf82189700 1 --2- 172.21.15.104:0/3125379452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c137e40 0x7fcf7c13a220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c09dc00 tx=0x7fcf74009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee00a2539f436ce7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6effd700 1 -- 172.21.15.104:0/363051274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 msgr2=0x7f7e7009c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6effd700 1 --2- 172.21.15.104:0/363051274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e7009c8e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e7009dce0 tx=0x7f7e6000b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1962fac8fd2c6e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 1 -- 172.21.15.104:0/3125379452 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7400e070 con 0x7fcf7c137e40 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: dump: 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.118 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 5 monclient: authenticate success, global_id 4263 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 1 -- 172.21.15.104:0/3125379452 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf74012430 con 0x7fcf7c137e40 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 4 set_mon_vals no callback set 2024-09-18T14:21:36.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 1 -- 172.21.15.104:0/3125379452 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf740127d0 con 0x7fcf7c137e40 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf80986700 10 monclient: dump: 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.120 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 Processor -- start 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 1 -- 172.21.15.104:0/363051274 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000dc40 con 0x7f7e7009c510 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: dump: 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.121 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 5 monclient: authenticate success, global_id 4261 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- start start 2024-09-18T14:21:36.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient: init 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 1 -- 172.21.15.104:0/363051274 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6000dda0 con 0x7f7e7009c510 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 1 -- 172.21.15.104:0/363051274 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60017670 con 0x7f7e7009c510 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e6dffb700 10 monclient: dump: 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.123 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 4 set_mon_vals no callback set 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 10 monclient: shutdown 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 1 -- 172.21.15.104:0/3125379452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c137e40 msgr2=0x7fcf7c13a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3125379452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c137e40 0x7fcf7c13a220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c09dc00 tx=0x7fcf74009780 comp rx=0 tx=0).stop 2024-09-18T14:21:36.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:36.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febe406f240 0x7febe4133fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): picked mon.noname-a con 0x7febe406f240 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe406f610 0x7febe411dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): picked mon.noname-c con 0x7febe406f610 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a2c30 0x7febe411e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): picked mon.noname-b con 0x7febe40a2c30 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe41217b0 con 0x7febe406f240 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4098af0 con 0x7febe40a2c30 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe41213d0 con 0x7febe406f610 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febebef9700 10 monclient(hunting): authenticate will time out at 817.087337s 2024-09-18T14:21:36.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 10 monclient: shutdown 2024-09-18T14:21:36.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 1 -- 172.21.15.104:0/363051274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 msgr2=0x7f7e7009c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 1 --2- 172.21.15.104:0/363051274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e7009c8e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e7009dce0 tx=0x7f7e6000b4e0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febe406f240 0x7febe4133fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe406f610 0x7febe411dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febe406f240 0x7febe4133fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50954/0 (socket says 172.21.15.104:50954) 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 1 -- 172.21.15.104:0/1022141844 learned_addr learned my addr 172.21.15.104:0/1022141844 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 10 monclient(hunting): get_auth_request con 0x7febe406f240 auth_method 0 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 10 monclient(hunting): get_auth_request con 0x7febe406f610 auth_method 0 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9494700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 1 -- 172.21.15.104:0/363051274 shutdown_connections 2024-09-18T14:21:36.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 1 --2- 172.21.15.104:0/363051274 >> v2:172.21.15.104:3300/0 conn(0x7f7e70098770 0x7f7e7009d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 1 --2- 172.21.15.104:0/363051274 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 0x7f7e7009ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 1 --2- 172.21.15.104:0/363051274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e7009c8e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f7e756ac700 1 -- 172.21.15.104:0/363051274 >> 172.21.15.104:0/363051274 conn(0x7f7e700a1290 msgr2=0x7f7e700a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febe9c95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febea496700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a2c30 0x7febe411e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febea496700 10 monclient(hunting): get_auth_request con 0x7febe40a2c30 auth_method 0 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febea496700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febea496700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7febea496700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f3b76a2c700 10 monclient: get_monmap_and_config 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f3b76a2c700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.092+0000 7f3b76a2c700 10 monclient: monmap: 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:36.098927+0000 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:36.098927+0000 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:36.131 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 -- 172.21.15.104:0/363051274 shutdown_connections 2024-09-18T14:21:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- 172.21.15.104:0/3125379452 shutdown_connections 2024-09-18T14:21:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3125379452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c137e40 0x7fcf7c13a220 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- 172.21.15.104:0/3125379452 >> 172.21.15.104:0/3125379452 conn(0x7fcf7c068160 msgr2=0x7fcf7c0a11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-18T14:21:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 -- 172.21.15.104:0/1022141844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a2c30 msgr2=0x7febe411e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a2c30 0x7febe411e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 -- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febe406f240 msgr2=0x7febe4133fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febe406f240 0x7febe4133fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 monclient: found mon.noname-c 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 -- 172.21.15.104:0/1022141844 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc009ae0 con 0x7febe406f610 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febea496700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a2c30 0x7febe411e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- 172.21.15.104:0/3125379452 shutdown_connections 2024-09-18T14:21:36.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- 172.21.15.104:0/3125379452 wait complete. 2024-09-18T14:21:36.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 -- 172.21.15.104:0/363051274 wait complete. 2024-09-18T14:21:36.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 1 Processor -- start 2024-09-18T14:21:36.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient: get_monmap_and_config 2024-09-18T14:21:36.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient: monmap: 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:36.099980+0000 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:36.099980+0000 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:36.135 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 1 -- start start 2024-09-18T14:21:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient: init 2024-09-18T14:21:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T14:21:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b700a1900 0x7f3b70131f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient(hunting): picked mon.noname-b con 0x7f3b700a1900 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3b7009ad20 0x7f3b7009b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient(hunting): picked mon.noname-c con 0x7f3b7009ad20 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient: monmap: 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b7009bbe0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b70099510 con 0x7f3b7009bbe0 2024-09-18T14:21:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b700a1cd0 con 0x7f3b7009ad20 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7009da10 con 0x7f3b700a1900 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b76a2c700 10 monclient(hunting): authenticate will time out at 817.088993s 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b7622b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3b7009bbe0 0x7f3b7009c060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47452/0 (socket says 172.21.15.104:47452) 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b7622b700 1 -- 172.21.15.104:0/1093522956 learned_addr learned my addr 172.21.15.104:0/1093522956 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 1 --2- 172.21.15.104:0/1093522956 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b700a1900 0x7f3b70131f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 1 Processor -- start 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): get_auth_request con 0x7f3b700a1900 auth_method 0 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient: monmap: 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 1 --2- 172.21.15.104:0/1093522956 >> v2:172.21.15.174:3300/0 conn(0x7f3b7009ad20 0x7f3b7009b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 10 monclient(hunting): get_auth_request con 0x7f3b7009ad20 auth_method 0 2024-09-18T14:21:36.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 Processor -- start 2024-09-18T14:21:36.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 Processor -- start 2024-09-18T14:21:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 1 -- start start 2024-09-18T14:21:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient: init 2024-09-18T14:21:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 -- start start 2024-09-18T14:21:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient: init 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75229700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 1 -- 172.21.15.104:0/1093522956 >> v2:172.21.15.174:3300/0 conn(0x7f3b7009ad20 msgr2=0x7f3b7009b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 1 --2- 172.21.15.104:0/1093522956 >> v2:172.21.15.174:3300/0 conn(0x7f3b7009ad20 0x7f3b7009b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 1 -- 172.21.15.104:0/1093522956 >> v1:172.21.15.104:6789/0 conn(0x7f3b7009bbe0 legacy=0x7f3b7009c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 1 monclient: found mon.noname-b 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f3b75a2a700 1 -- 172.21.15.104:0/1093522956 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7009b630 con 0x7f3b700a1900 2024-09-18T14:21:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- start start 2024-09-18T14:21:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient: init 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 0x7f7e70134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): picked mon.noname-c con 0x7f7e70095fa0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 0x7f7e70141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): picked mon.noname-a con 0x7f7e70098770 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e70141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): picked mon.noname-b con 0x7f7e7009c510 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70141ed0 con 0x7f7e70098770 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70122a90 con 0x7f7e7009c510 2024-09-18T14:21:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70096660 con 0x7f7e70095fa0 2024-09-18T14:21:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e756ac700 10 monclient(hunting): authenticate will time out at 817.090175s 2024-09-18T14:21:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T14:21:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 0x7f7e70141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6b5009bfd0 0x7f6b5009b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient(hunting): picked mon.noname-c con 0x7f6b5009bfd0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:36.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b5009ff00 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:36.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b500a0a10 0x7f6b501396a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient(hunting): picked mon.noname-a con 0x7f6b500a0a10 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:36.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5009c3a0 con 0x7f6b5009ff00 2024-09-18T14:21:36.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5009a610 con 0x7f6b500a0a10 2024-09-18T14:21:36.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b500a1b00 con 0x7f6b5009bfd0 2024-09-18T14:21:36.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 0x7f7e70141450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50958/0 (socket says 172.21.15.104:50958) 2024-09-18T14:21:36.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:36.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b5830c700 10 monclient(hunting): authenticate will time out at 817.090310s 2024-09-18T14:21:36.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 -- 172.21.15.104:0/2760832422 learned_addr learned my addr 172.21.15.104:0/2760832422 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): get_auth_request con 0x7f7e70098770 auth_method 0 2024-09-18T14:21:36.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 0x7fcf7c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): picked mon.noname-b con 0x7fcf7c09ad20 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c0a1900 0x7fcf7c137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): picked mon.noname-a con 0x7fcf7c0a1900 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcf7c137e40 0x7fcf7c132c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): picked mon.noname-c con 0x7fcf7c137e40 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c135a20 con 0x7fcf7c0a1900 2024-09-18T14:21:36.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c097c70 con 0x7fcf7c09ad20 2024-09-18T14:21:36.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c135740 con 0x7fcf7c137e40 2024-09-18T14:21:36.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf8298a700 10 monclient(hunting): authenticate will time out at 817.090470s 2024-09-18T14:21:36.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b500a0a10 0x7f6b501396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b500a0a10 0x7f6b501396a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50960/0 (socket says 172.21.15.104:50960) 2024-09-18T14:21:36.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 -- 172.21.15.104:0/1616027824 learned_addr learned my addr 172.21.15.104:0/1616027824 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): get_auth_request con 0x7f6b500a0a10 auth_method 0 2024-09-18T14:21:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6f7fe700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e70141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6f7fe700 10 monclient(hunting): get_auth_request con 0x7f7e7009c510 auth_method 0 2024-09-18T14:21:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 0x7fcf7c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 0x7fcf7c06ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.104:38754/0 (socket says 172.21.15.104:38754) 2024-09-18T14:21:36.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 -- 172.21.15.104:0/3314384216 learned_addr learned my addr 172.21.15.104:0/3314384216 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): get_auth_request con 0x7fcf7c09ad20 auth_method 0 2024-09-18T14:21:36.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b560a8700 1 --2- 172.21.15.104:0/1616027824 >> v2:172.21.15.174:3300/0 conn(0x7f6b5009bfd0 0x7f6b5009b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b560a8700 10 monclient(hunting): get_auth_request con 0x7f6b5009bfd0 auth_method 0 2024-09-18T14:21:36.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b560a8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b560a8700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b560a8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 -- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 msgr2=0x7f7e70134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 0x7f7e70134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 -- 172.21.15.104:0/1616027824 >> v2:172.21.15.174:3300/0 conn(0x7f6b5009bfd0 msgr2=0x7f6b5009b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 --2- 172.21.15.104:0/1616027824 >> v2:172.21.15.174:3300/0 conn(0x7f6b5009bfd0 0x7f6b5009b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 -- 172.21.15.104:0/2760832422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 msgr2=0x7f7e70141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e70141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 monclient: found mon.noname-a 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6e7fc700 1 -- 172.21.15.104:0/2760832422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6000b780 con 0x7f7e70098770 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f7e6f7fe700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e70141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 -- 172.21.15.104:0/1616027824 >> v1:172.21.15.174:6789/0 conn(0x7f6b5009ff00 legacy=0x7f6b500a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 monclient: found mon.noname-a 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7f6b568a9700 1 -- 172.21.15.104:0/1616027824 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5009a330 con 0x7f6b500a0a10 2024-09-18T14:21:36.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2024-09-18T14:21:36.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 -- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe406f610 msgr2=0x7febe411dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febe9494700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe406f610 0x7febe411dae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7febd400c4e0 tx=0x7febd4007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 -- 172.21.15.104:0/3314384216 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcf7c137e40 msgr2=0x7fcf7c132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcf7c137e40 0x7fcf7c132c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 -- 172.21.15.104:0/3314384216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c0a1900 msgr2=0x7fcf7c137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c0a1900 0x7fcf7c137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 monclient: found mon.noname-b 2024-09-18T14:21:36.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:21:36.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7fcf81988700 1 -- 172.21.15.104:0/3314384216 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74009a20 con 0x7fcf7c09ad20 2024-09-18T14:21:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 1 -- 172.21.15.104:0/1022141844 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd40117a0 con 0x7febe406f610 2024-09-18T14:21:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: dump: 2024-09-18T14:21:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febebef9700 5 monclient: authenticate success, global_id 4268 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 1 -- 172.21.15.104:0/1022141844 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd4011920 con 0x7febe406f610 2024-09-18T14:21:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 1 -- 172.21.15.104:0/1022141844 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4011ce0 con 0x7febe406f610 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febebef9700 10 monclient: _renew_subs 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febdb7fe700 4 set_mon_vals no callback set 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febebef9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe4142120 con 0x7febe406f610 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febd9ffb700 10 monclient: dump: 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.164 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febebef9700 10 monclient: _renew_subs 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febebef9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.096+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe4120200 con 0x7febe406f610 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b75a2a700 1 -- 172.21.15.104:0/1093522956 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b700a1900 msgr2=0x7f3b70131f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b75a2a700 1 --2- 172.21.15.104:0/1093522956 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b700a1900 0x7f3b70131f00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3b7009b820 tx=0x7f3b600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=368ee36563ab671a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 1 -- 172.21.15.104:0/1093522956 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60011840 con 0x7f3b700a1900 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: dump: 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 1 -- 172.21.15.104:0/1093522956 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b600119a0 con 0x7f3b700a1900 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 1 -- 172.21.15.104:0/1093522956 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60011d40 con 0x7f3b700a1900 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b74a28700 10 monclient: dump: 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.168 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 5 monclient: authenticate success, global_id 4271 2024-09-18T14:21:36.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:36.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 4 set_mon_vals no callback set 2024-09-18T14:21:36.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febd9ffb700 1 -- 172.21.15.104:0/1022141844 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7febd4012630 con 0x7febe406f610 2024-09-18T14:21:36.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febebef9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febebef9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febebef9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe411f680 con 0x7febe406f610 2024-09-18T14:21:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient: shutdown 2024-09-18T14:21:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- 172.21.15.104:0/1093522956 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b700a1900 msgr2=0x7f3b70131f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/1093522956 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b700a1900 0x7f3b70131f00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3b7009b820 tx=0x7f3b600091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febd9ffb700 1 --2- 172.21.15.104:0/1022141844 >> v2:172.21.15.174:6812/17603 conn(0x7febc8035ad0 0x7febc8037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e6e7fc700 1 -- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 msgr2=0x7f7e70141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e6e7fc700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 0x7f7e70141450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400ecb0 tx=0x7f7e640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febd9ffb700 1 -- 172.21.15.104:0/1022141844 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7febd4047af0 con 0x7febe406f610 2024-09-18T14:21:36.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b568a9700 1 -- 172.21.15.104:0/1616027824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 msgr2=0x7f6b501396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b568a9700 1 --2- 172.21.15.104:0/1616027824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b501396a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009f20 tx=0x7f6b48009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accc21b06841ccf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- 172.21.15.104:0/1093522956 shutdown_connections 2024-09-18T14:21:36.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/1093522956 >> v2:172.21.15.174:3300/0 conn(0x7f3b7009ad20 0x7f3b7009b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/1093522956 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b700a1900 0x7f3b70131f00 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3b7009b820 tx=0x7f3b600091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- 172.21.15.104:0/1093522956 >> 172.21.15.104:0/1093522956 conn(0x7f3b70068160 msgr2=0x7f3b700a11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febe9c95700 1 --2- 172.21.15.104:0/1022141844 >> v2:172.21.15.174:6812/17603 conn(0x7febc8035ad0 0x7febc8037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7febe9c95700 10 monclient: get_auth_request con 0x7febc8035ad0 auth_method 0 2024-09-18T14:21:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 1 -- 172.21.15.104:0/1616027824 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800e070 con 0x7f6b500a0a10 2024-09-18T14:21:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- 172.21.15.104:0/1093522956 shutdown_connections 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: dump: 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.174 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- 172.21.15.104:0/1093522956 wait complete. 2024-09-18T14:21:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 1 -- 172.21.15.104:0/1616027824 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4800bc50 con 0x7f6b500a0a10 2024-09-18T14:21:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 1 -- 172.21.15.104:0/1616027824 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b480092e0 con 0x7f6b500a0a10 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b550a6700 10 monclient: dump: 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.176 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf81988700 1 -- 172.21.15.104:0/3314384216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 msgr2=0x7fcf7c06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf81988700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 0x7fcf7c06ec60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000c5e0 tx=0x7fcf70004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f227d9cf700 10 monclient: get_monmap_and_config 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f227d9cf700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f227d9cf700 10 monclient: monmap: 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:36.105439+0000 2024-09-18T14:21:36.177 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:36.105439+0000 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 1 -- 172.21.15.104:0/3314384216 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70005d40 con 0x7fcf7c09ad20 2024-09-18T14:21:36.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf81988700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 0x7fcf7c06ec60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000c5e0 tx=0x7fcf70004a80 comp rx=0 tx=0).queue_received will delay after 1.72667e+09 on 0x7fcf700070d0 config(0 keys) v1 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 10 monclient: dump: 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf8298a700 5 monclient: authenticate success, global_id 4273 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 1 -- 172.21.15.104:0/3314384216 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf700070d0 con 0x7fcf7c09ad20 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf8298a700 10 monclient: _renew_subs 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf8298a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7c135d00 con 0x7fcf7c09ad20 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf6f7fe700 4 set_mon_vals no callback set 2024-09-18T14:21:36.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient: monmap: 2024-09-18T14:21:36.181 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.181 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:36.181 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:36.181 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:36.181 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.181 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.182 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:36.182 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:36.182 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:36.182 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b5830c700 5 monclient: authenticate success, global_id 4275 2024-09-18T14:21:36.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b5830c700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:36.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b5830c700 4 set_mon_vals no callback set 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 1 -- 172.21.15.104:0/2760832422 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400cb80 con 0x7f7e70098770 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: dump: 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.183 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.184 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.184 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.184 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 1 -- 172.21.15.104:0/2760832422 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6400cce0 con 0x7f7e70098770 2024-09-18T14:21:36.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 Processor -- start 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 5 monclient: authenticate success, global_id 4272 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 10 monclient: _renew_subs 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e70121540 con 0x7f7e70098770 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 1 -- 172.21.15.104:0/2760832422 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e640136c0 con 0x7f7e70098770 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e56ffd700 10 monclient: dump: 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.185 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e6d7fa700 4 set_mon_vals no callback set 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 10 monclient: _renew_subs 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e70134850 con 0x7f7e70098770 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e70064b30 con 0x7f7e70098770 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf8298a700 10 monclient: _renew_subs 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf8298a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7c097730 con 0x7fcf7c09ad20 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b5830c700 10 monclient: shutdown 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b5830c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b5830c700 1 -- 172.21.15.104:0/1616027824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 msgr2=0x7f6b501396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f6b5830c700 1 --2- 172.21.15.104:0/1616027824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b501396a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009f20 tx=0x7f6b48009840 comp rx=0 tx=0).stop 2024-09-18T14:21:36.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- start start 2024-09-18T14:21:36.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient: init 2024-09-18T14:21:36.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7009ad20 0x7f3b70141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): picked mon.noname-b con 0x7f3b7009ad20 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b7009bbe0 0x7f3b7012eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): picked mon.noname-a con 0x7f3b7009bbe0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 0x7f3b7012f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): picked mon.noname-c con 0x7f3b701419a0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7012d430 con 0x7f3b7009bbe0 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b70130f00 con 0x7f3b7009ad20 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b701423c0 con 0x7f3b701419a0 2024-09-18T14:21:36.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.100+0000 7f3b76a2c700 10 monclient(hunting): authenticate will time out at 817.095989s 2024-09-18T14:21:36.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- 172.21.15.104:0/1616027824 shutdown_connections 2024-09-18T14:21:36.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 --2- 172.21.15.104:0/1616027824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b501396a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 --2- 172.21.15.104:0/1616027824 >> v2:172.21.15.174:3300/0 conn(0x7f6b5009bfd0 0x7f6b5009b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- 172.21.15.104:0/1616027824 >> 172.21.15.104:0/1616027824 conn(0x7f6b50069500 msgr2=0x7f6b500a5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f7e56ffd700 1 -- 172.21.15.104:0/2760832422 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f7e64013a70 con 0x7f7e70098770 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf6dffb700 1 -- 172.21.15.104:0/3314384216 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70007490 con 0x7fcf7c09ad20 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf6dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf6dffb700 10 monclient: dump: 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.191 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75a2a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7009ad20 0x7f3b70141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 0x7f3b7012f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75a2a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7009ad20 0x7f3b70141460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.104:38774/0 (socket says 172.21.15.104:38774) 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75a2a700 1 -- 172.21.15.104:0/3264183619 learned_addr learned my addr 172.21.15.104:0/3264183619 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 0x7f3b7012f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35840/0 (socket says 172.21.15.104:35840) 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75a2a700 10 monclient(hunting): get_auth_request con 0x7f3b7009ad20 auth_method 0 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75a2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75a2a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75a2a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): get_auth_request con 0x7f3b701419a0 auth_method 0 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f7e56ffd700 1 --2- 172.21.15.104:0/2760832422 >> v2:172.21.15.174:6812/17603 conn(0x7f7e5c035ad0 0x7f7e5c037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f7e56ffd700 1 -- 172.21.15.104:0/2760832422 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f7e64048ad0 con 0x7f7e70098770 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- 172.21.15.104:0/1616027824 shutdown_connections 2024-09-18T14:21:36.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b7009bbe0 0x7f3b7012eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 10 monclient(hunting): get_auth_request con 0x7f3b7009bbe0 auth_method 0 2024-09-18T14:21:36.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 1 Processor -- start 2024-09-18T14:21:36.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b75229700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 1 -- start start 2024-09-18T14:21:36.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient: init 2024-09-18T14:21:36.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f7e6effd700 1 --2- 172.21.15.104:0/2760832422 >> v2:172.21.15.174:6812/17603 conn(0x7f7e5c035ad0 0x7f7e5c037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f7e6effd700 10 monclient: get_auth_request con 0x7f7e5c035ad0 auth_method 0 2024-09-18T14:21:36.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T14:21:36.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- 172.21.15.104:0/1616027824 wait complete. 2024-09-18T14:21:36.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227806f140 0x7f2278097e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient(hunting): picked mon.noname-b con 0x7f227806f140 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f227806a3b0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:36.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f227809af20 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:36.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf6dffb700 1 -- 172.21.15.104:0/3314384216 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fcf7001b3b0 con 0x7fcf7c09ad20 2024-09-18T14:21:36.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf6dffb700 1 --2- 172.21.15.104:0/3314384216 >> v2:172.21.15.174:6812/17603 conn(0x7fcf60035a80 0x7fcf60037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22780971e0 con 0x7f227806a3b0 2024-09-18T14:21:36.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227809db00 con 0x7f227809af20 2024-09-18T14:21:36.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227809c390 con 0x7f227806f140 2024-09-18T14:21:36.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f227d9cf700 10 monclient(hunting): authenticate will time out at 817.097180s 2024-09-18T14:21:36.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f22767fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f227806a3b0 0x7f227809c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47466/0 (socket says 172.21.15.104:47466) 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f22767fc700 1 -- 172.21.15.104:0/1547437200 learned_addr learned my addr 172.21.15.104:0/1547437200 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 1 --2- 172.21.15.104:0/1547437200 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227806f140 0x7f2278097e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): get_auth_request con 0x7f227806f140 auth_method 0 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf81187700 1 --2- 172.21.15.104:0/3314384216 >> v2:172.21.15.174:6812/17603 conn(0x7fcf60035a80 0x7fcf60037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient: monmap: 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.200 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf81187700 10 monclient: get_auth_request con 0x7fcf60035a80 auth_method 0 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf8298a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf8298a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf8298a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7c063450 con 0x7fcf7c09ad20 2024-09-18T14:21:36.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7febe9c95700 1 --2- 172.21.15.104:0/1022141844 >> v2:172.21.15.174:6812/17603 conn(0x7febc8035ad0 0x7febc8037f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febdc009450 tx=0x7febdc017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-09-18T14:21:36.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 -- 172.21.15.104:0/3264183619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7009ad20 msgr2=0x7f3b70141460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7009ad20 0x7f3b70141460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 -- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b7009bbe0 msgr2=0x7f3b7012eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b7009bbe0 0x7f3b7012eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 monclient: found mon.noname-c 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 -- 172.21.15.104:0/3264183619 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b60009420 con 0x7f3b701419a0 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 1 -- 172.21.15.104:0/1547437200 >> v1:172.21.15.174:6789/0 conn(0x7f227809af20 legacy=0x7f227809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 1 -- 172.21.15.104:0/1547437200 >> v1:172.21.15.104:6789/0 conn(0x7f227806a3b0 legacy=0x7f227809c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 1 monclient: found mon.noname-b 2024-09-18T14:21:36.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f2276ffd700 1 -- 172.21.15.104:0/1547437200 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2278095c90 con 0x7f227806f140 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 Processor -- start 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- start start 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient: init 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6b5009bfd0 0x7f6b50098660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): picked mon.noname-c con 0x7f6b5009bfd0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5009ff00 0x7f6b50098ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): picked mon.noname-b con 0x7f6b5009ff00 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b5012d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): picked mon.noname-a con 0x7f6b500a0a10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b500a0f10 con 0x7f6b500a0a10 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5009c900 con 0x7f6b5009ff00 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b50131c20 con 0x7f6b5009bfd0 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b5830c700 10 monclient(hunting): authenticate will time out at 817.098920s 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b5012d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b5012d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50984/0 (socket says 172.21.15.104:50984) 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 -- 172.21.15.104:0/2968188895 learned_addr learned my addr 172.21.15.104:0/2968188895 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): get_auth_request con 0x7f6b500a0a10 auth_method 0 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b558a7700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5009ff00 0x7f6b50098ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b558a7700 10 monclient(hunting): get_auth_request con 0x7f6b5009ff00 auth_method 0 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b558a7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b558a7700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b558a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-18T14:21:36.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 -- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6b5009bfd0 msgr2=0x7f6b50098660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6b5009bfd0 0x7f6b50098660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 -- 172.21.15.104:0/2968188895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5009ff00 msgr2=0x7f6b50098ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5009ff00 0x7f6b50098ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 monclient: found mon.noname-a 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f6b568a9700 1 -- 172.21.15.104:0/2968188895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b48009ae0 con 0x7f6b500a0a10 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 -- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 msgr2=0x7f3b7012f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f3b7622b700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 0x7f3b7012f080 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00c510 tx=0x7f3b6c007f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.104+0000 7f7e6effd700 1 --2- 172.21.15.104:0/2760832422 >> v2:172.21.15.174:6812/17603 conn(0x7f7e5c035ad0 0x7f7e5c037f80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7e7009dce0 tx=0x7f7e6001a000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2276ffd700 1 -- 172.21.15.104:0/1547437200 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227806f140 msgr2=0x7f2278097e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2276ffd700 1 --2- 172.21.15.104:0/1547437200 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227806f140 0x7f2278097e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f226c009910 tx=0x7f226c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35138c935896d6e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 1 -- 172.21.15.104:0/3264183619 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c005d40 con 0x7f3b701419a0 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: dump: 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.211 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 1 -- 172.21.15.104:0/3264183619 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c0070d0 con 0x7f3b701419a0 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 1 -- 172.21.15.104:0/3264183619 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c007490 con 0x7f3b701419a0 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 5 monclient: authenticate success, global_id 4274 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 10 monclient: _renew_subs 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 1 -- 172.21.15.104:0/1547437200 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c00ba70 con 0x7f227806f140 2024-09-18T14:21:36.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7012d050 con 0x7f3b701419a0 2024-09-18T14:21:36.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:36.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7febd9ffb700 1 -- 172.21.15.104:0/1022141844 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7febd4011e40 con 0x7febe406f610 2024-09-18T14:21:36.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7febd9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7febd9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: dump: 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.214 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 10 monclient: dump: 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b67fff700 4 set_mon_vals no callback set 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 10 monclient: _renew_subs 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7012be10 con 0x7f3b701419a0 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 1 -- 172.21.15.104:0/1547437200 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226c00bbd0 con 0x7f227806f140 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 1 -- 172.21.15.104:0/1547437200 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c011420 con 0x7f227806f140 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f2275ffb700 10 monclient: dump: 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 5 monclient: authenticate success, global_id 4277 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 4 set_mon_vals no callback set 2024-09-18T14:21:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7fcf81187700 1 --2- 172.21.15.104:0/3314384216 >> v2:172.21.15.174:6812/17603 conn(0x7fcf60035a80 0x7fcf60037f30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcf78007fd0 tx=0x7fcf78007b20 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b568a9700 1 -- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 msgr2=0x7f6b5012d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b568a9700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b5012d4c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6b5006b430 tx=0x7f6b48017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 10 monclient: shutdown 2024-09-18T14:21:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 DispatchQueue::entry inject delay of 0.002000 2024-09-18T14:21:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 -- 172.21.15.104:0/1547437200 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227806f140 msgr2=0x7f2278097e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 --2- 172.21.15.104:0/1547437200 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227806f140 0x7f2278097e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f226c009910 tx=0x7f226c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 -- 172.21.15.104:0/1547437200 shutdown_connections 2024-09-18T14:21:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 --2- 172.21.15.104:0/1547437200 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227806f140 0x7f2278097e60 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f226c009910 tx=0x7f226c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 -- 172.21.15.104:0/1547437200 >> 172.21.15.104:0/1547437200 conn(0x7f22780a1290 msgr2=0x7f22780a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7012ce50 con 0x7f3b701419a0 2024-09-18T14:21:36.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f7e56ffd700 1 -- 172.21.15.104:0/2760832422 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7e6400ce50 con 0x7f7e70098770 2024-09-18T14:21:36.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f7e56ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f7e56ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:36.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 -- 172.21.15.104:0/1547437200 shutdown_connections 2024-09-18T14:21:36.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 -- 172.21.15.104:0/1547437200 wait complete. 2024-09-18T14:21:36.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 1 -- 172.21.15.104:0/3264183619 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f3b6c01b3b0 con 0x7f3b701419a0 2024-09-18T14:21:36.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 10 monclient: build_initial_monmap 2024-09-18T14:21:36.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 10 monclient: monmap: 2024-09-18T14:21:36.222 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:36.222 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:36.222 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 1 --2- 172.21.15.104:0/3264183619 >> v2:172.21.15.174:6812/17603 conn(0x7f3b54035b00 0x7f3b54037fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b65ffb700 1 -- 172.21.15.104:0/3264183619 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f3b6c047680 con 0x7f3b701419a0 2024-09-18T14:21:36.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b75a2a700 1 --2- 172.21.15.104:0/3264183619 >> v2:172.21.15.174:6812/17603 conn(0x7f3b54035b00 0x7f3b54037fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f3b75a2a700 10 monclient: get_auth_request con 0x7f3b54035b00 auth_method 0 2024-09-18T14:21:36.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 Processor -- start 2024-09-18T14:21:36.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 1 -- start start 2024-09-18T14:21:36.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f227d9cf700 10 monclient: init 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48012c90 con 0x7f6b500a0a10 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: dump: 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.225 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b5830c700 5 monclient: authenticate success, global_id 4281 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48012df0 con 0x7f6b500a0a10 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4801d670 con 0x7f6b500a0a10 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b467fc700 10 monclient: dump: 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.108+0000 7f6b47fff700 4 set_mon_vals no callback set 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f227806a3b0 0x7f2278141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): picked mon.noname-b con 0x7f227806a3b0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 0x7f227813a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): picked mon.noname-c con 0x7f227809af20 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22781346e0 0x7f22781321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): picked mon.noname-a con 0x7f22781346e0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780998a0 con 0x7f22781346e0 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2278135640 con 0x7f227806a3b0 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2278135920 con 0x7f227809af20 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f227d9cf700 10 monclient(hunting): authenticate will time out at 817.104310s 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 0x7f227813a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 0x7f227813a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35860/0 (socket says 172.21.15.104:35860) 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 -- 172.21.15.104:0/702172936 learned_addr learned my addr 172.21.15.104:0/702172936 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): get_auth_request con 0x7f227809af20 auth_method 0 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f227806a3b0 0x7f2278141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 10 monclient(hunting): get_auth_request con 0x7f227806a3b0 auth_method 0 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f2276ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-18T14:21:36.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 -- 172.21.15.104:0/702172936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f227806a3b0 msgr2=0x7f2278141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f227806a3b0 0x7f2278141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 -- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22781346e0 msgr2=0x7f22781321d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22781346e0 0x7f22781321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient: _finish_hunting 0 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 monclient: found mon.noname-c 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 -- 172.21.15.104:0/702172936 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c009420 con 0x7f227809af20 2024-09-18T14:21:36.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 10 monclient: _renew_subs 2024-09-18T14:21:36.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b50130f50 con 0x7f6b500a0a10 2024-09-18T14:21:36.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 10 monclient: _renew_subs 2024-09-18T14:21:36.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b50130990 con 0x7f6b500a0a10 2024-09-18T14:21:36.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5012ef00 con 0x7f6b500a0a10 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f6b4801da20 con 0x7f6b500a0a10 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b467fc700 1 --2- 172.21.15.104:0/2968188895 >> v2:172.21.15.174:6812/17603 conn(0x7f6b34035ad0 0x7f6b34037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f6b48049620 con 0x7f6b500a0a10 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b560a8700 1 --2- 172.21.15.104:0/2968188895 >> v2:172.21.15.174:6812/17603 conn(0x7f6b34035ad0 0x7f6b34037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f6b560a8700 10 monclient: get_auth_request con 0x7f6b34035ad0 auth_method 0 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 -- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 msgr2=0x7f227813a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:36.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f22767fc700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 0x7f227813a440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f227000cca0 tx=0x7f2270005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.112+0000 7f3b75a2a700 1 --2- 172.21.15.104:0/3264183619 >> v2:172.21.15.174:6812/17603 conn(0x7f3b54035b00 0x7f3b54037fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006f80 tx=0x7f3b60017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 1 -- 172.21.15.104:0/702172936 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227000abd0 con 0x7f227809af20 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: dump: 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.236 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: _finish_auth 0 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 5 monclient: authenticate success, global_id 4280 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 1 -- 172.21.15.104:0/702172936 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227000ad50 con 0x7f227809af20 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 10 monclient: _renew_subs 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2278133b30 con 0x7f227809af20 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 1 -- 172.21.15.104:0/702172936 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2270007340 con 0x7f227809af20 2024-09-18T14:21:36.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 10 monclient: dump: 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 10 monclient: _renew_subs 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22781423d0 con 0x7f227809af20 2024-09-18T14:21:36.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f22757fa700 4 set_mon_vals no callback set 2024-09-18T14:21:36.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 1 -- 172.21.15.104:0/702172936 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f2270007730 con 0x7f227809af20 2024-09-18T14:21:36.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f6b560a8700 1 --2- 172.21.15.104:0/2968188895 >> v2:172.21.15.174:6812/17603 conn(0x7f6b34035ad0 0x7f6b34037f80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009a60 tx=0x7f6b40008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f3b65ffb700 1 -- 172.21.15.104:0/3264183619 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3b6c01b660 con 0x7f3b701419a0 2024-09-18T14:21:36.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f3b65ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f3b65ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:36.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2278064b30 con 0x7f227809af20 2024-09-18T14:21:36.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7f225effd700 1 --2- 172.21.15.104:0/702172936 >> v2:172.21.15.174:6812/17603 conn(0x7f2264035ad0 0x7f2264037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:36.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.116+0000 7fcf6dffb700 1 -- 172.21.15.104:0/3314384216 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fcf7004b120 con 0x7fcf7c09ad20 2024-09-18T14:21:36.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.120+0000 7f2276ffd700 1 --2- 172.21.15.104:0/702172936 >> v2:172.21.15.174:6812/17603 conn(0x7f2264035ad0 0x7f2264037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:36.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.120+0000 7f2276ffd700 10 monclient: get_auth_request con 0x7f2264035ad0 auth_method 0 2024-09-18T14:21:36.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.120+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6b48007550 con 0x7f6b500a0a10 2024-09-18T14:21:36.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.120+0000 7f6b467fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.120+0000 7f6b467fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:36.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.124+0000 7f225effd700 1 -- 172.21.15.104:0/702172936 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f2270048450 con 0x7f227809af20 2024-09-18T14:21:36.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.124+0000 7f2276ffd700 1 --2- 172.21.15.104:0/702172936 >> v2:172.21.15.174:6812/17603 conn(0x7f2264035ad0 0x7f2264037f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2278069ac0 tx=0x7f226c016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:36.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.124+0000 7f225effd700 1 -- 172.21.15.104:0/702172936 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f22700079e0 con 0x7f227809af20 2024-09-18T14:21:36.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.124+0000 7f225effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.124+0000 7f225effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:36.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.276+0000 7f3b76a2c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T14:21:36.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.276+0000 7f3b76a2c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T14:21:36.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.276+0000 7f3b76a2c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.276+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3b70063450 con 0x7f3b701419a0 2024-09-18T14:21:36.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b65ffb700 1 -- 172.21.15.104:0/3264183619 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b6c0192c0 con 0x7f3b701419a0 2024-09-18T14:21:36.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b65ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T14:21:36.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b65ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:36.287 INFO:teuthology.orchestra.run.smithi104.stdout:47244640258 2024-09-18T14:21:36.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 >> v2:172.21.15.174:6812/17603 conn(0x7f3b54035b00 msgr2=0x7f3b54037fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/3264183619 >> v2:172.21.15.174:6812/17603 conn(0x7f3b54035b00 0x7f3b54037fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006f80 tx=0x7f3b60017000 comp rx=0 tx=0).stop 2024-09-18T14:21:36.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 10 monclient: shutdown 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 msgr2=0x7f3b7012f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 0x7f3b7012f080 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00c510 tx=0x7f3b6c007f70 comp rx=0 tx=0).stop 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 shutdown_connections 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/3264183619 >> v2:172.21.15.174:6812/17603 conn(0x7f3b54035b00 0x7f3b54037fb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3b701419a0 0x7f3b7012f080 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b7009bbe0 0x7f3b7012eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 --2- 172.21.15.104:0/3264183619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7009ad20 0x7f3b70141460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 >> 172.21.15.104:0/3264183619 conn(0x7f3b70068160 msgr2=0x7f3b70099ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 shutdown_connections 2024-09-18T14:21:36.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.280+0000 7f3b76a2c700 1 -- 172.21.15.104:0/3264183619 wait complete. 2024-09-18T14:21:36.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.0 2024-09-18T14:21:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7febebef9700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T14:21:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7febebef9700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T14:21:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7febebef9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7febe40636d0 con 0x7febe406f610 2024-09-18T14:21:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7febd9ffb700 1 -- 172.21.15.104:0/1022141844 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7febd40102e0 con 0x7febe406f610 2024-09-18T14:21:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7febd9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T14:21:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7febd9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:36.305 INFO:teuthology.orchestra.run.smithi104.stdout:47244640258 2024-09-18T14:21:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7f7e756ac700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T14:21:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7f7e756ac700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T14:21:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7f7e756ac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.296+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7e7009d680 con 0x7f7e70098770 2024-09-18T14:21:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e56ffd700 1 -- 172.21.15.104:0/2760832422 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e640117c0 con 0x7f7e70098770 2024-09-18T14:21:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e56ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T14:21:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e56ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:36.307 INFO:teuthology.orchestra.run.smithi104.stdout:47244640258 2024-09-18T14:21:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 >> v2:172.21.15.174:6812/17603 conn(0x7febc8035ad0 msgr2=0x7febc8037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 --2- 172.21.15.104:0/1022141844 >> v2:172.21.15.174:6812/17603 conn(0x7febc8035ad0 0x7febc8037f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febdc009450 tx=0x7febdc017000 comp rx=0 tx=0).stop 2024-09-18T14:21:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 10 monclient: shutdown 2024-09-18T14:21:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe406f610 msgr2=0x7febe411dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe406f610 0x7febe411dae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7febd400c4e0 tx=0x7febd4007ea0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 shutdown_connections 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 --2- 172.21.15.104:0/1022141844 >> v2:172.21.15.174:6812/17603 conn(0x7febc8035ad0 0x7febc8037f80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a2c30 0x7febe411e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7febe406f610 0x7febe411dae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 --2- 172.21.15.104:0/1022141844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febe406f240 0x7febe4133fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 >> 172.21.15.104:0/1022141844 conn(0x7febe4068f60 msgr2=0x7febe4097a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 shutdown_connections 2024-09-18T14:21:36.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7febebef9700 1 -- 172.21.15.104:0/1022141844 wait complete. 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 >> v2:172.21.15.174:6812/17603 conn(0x7f7e5c035ad0 msgr2=0x7f7e5c037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 --2- 172.21.15.104:0/2760832422 >> v2:172.21.15.174:6812/17603 conn(0x7f7e5c035ad0 0x7f7e5c037f80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7e7009dce0 tx=0x7f7e6001a000 comp rx=0 tx=0).stop 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 10 monclient: shutdown 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 msgr2=0x7f7e70141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 0x7f7e70141450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400ecb0 tx=0x7f7e640091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 shutdown_connections 2024-09-18T14:21:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 --2- 172.21.15.104:0/2760832422 >> v2:172.21.15.174:6812/17603 conn(0x7f7e5c035ad0 0x7f7e5c037f80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e7009c510 0x7f7e70141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e70098770 0x7f7e70141450 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400ecb0 tx=0x7f7e640091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 --2- 172.21.15.104:0/2760832422 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7e70095fa0 0x7f7e70134310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 >> 172.21.15.104:0/2760832422 conn(0x7f7e700a1290 msgr2=0x7f7e7009f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 shutdown_connections 2024-09-18T14:21:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.300+0000 7f7e756ac700 1 -- 172.21.15.104:0/2760832422 wait complete. 2024-09-18T14:21:36.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-18T14:21:36.320 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.1 2024-09-18T14:21:36.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.392+0000 7f227d9cf700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T14:21:36.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.392+0000 7f227d9cf700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T14:21:36.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.392+0000 7f227d9cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:36.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.392+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f227809a670 con 0x7f227809af20 2024-09-18T14:21:36.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.392+0000 7f225effd700 1 -- 172.21.15.104:0/702172936 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2270010300 con 0x7f227809af20 2024-09-18T14:21:36.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.392+0000 7f225effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T14:21:36.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.392+0000 7f225effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:36.401 INFO:teuthology.orchestra.run.smithi104.stdout:47244640258 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 >> v2:172.21.15.174:6812/17603 conn(0x7f2264035ad0 msgr2=0x7f2264037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 --2- 172.21.15.104:0/702172936 >> v2:172.21.15.174:6812/17603 conn(0x7f2264035ad0 0x7f2264037f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2278069ac0 tx=0x7f226c016000 comp rx=0 tx=0).stop 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 10 monclient: shutdown 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 msgr2=0x7f227813a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 0x7f227813a440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f227000cca0 tx=0x7f2270005480 comp rx=0 tx=0).stop 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 shutdown_connections 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 --2- 172.21.15.104:0/702172936 >> v2:172.21.15.174:6812/17603 conn(0x7f2264035ad0 0x7f2264037f80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22781346e0 0x7f22781321d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f227809af20 0x7f227813a440 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 --2- 172.21.15.104:0/702172936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f227806a3b0 0x7f2278141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 >> 172.21.15.104:0/702172936 conn(0x7f22780a1290 msgr2=0x7f227809bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 shutdown_connections 2024-09-18T14:21:36.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f227d9cf700 1 -- 172.21.15.104:0/702172936 wait complete. 2024-09-18T14:21:36.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f6b5830c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T14:21:36.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f6b5830c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T14:21:36.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f6b5830c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:36.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6b500636d0 con 0x7f6b500a0a10 2024-09-18T14:21:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f6b467fc700 1 -- 172.21.15.104:0/2968188895 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b48026bb0 con 0x7f6b500a0a10 2024-09-18T14:21:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f6b467fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T14:21:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.396+0000 7f6b467fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:36.406 INFO:teuthology.orchestra.run.smithi104.stdout:47244640258 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 >> v2:172.21.15.174:6812/17603 conn(0x7f6b34035ad0 msgr2=0x7f6b34037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 --2- 172.21.15.104:0/2968188895 >> v2:172.21.15.174:6812/17603 conn(0x7f6b34035ad0 0x7f6b34037f80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009a60 tx=0x7f6b40008040 comp rx=0 tx=0).stop 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 10 monclient: shutdown 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 msgr2=0x7f6b5012d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b5012d4c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6b5006b430 tx=0x7f6b48017000 comp rx=0 tx=0).stop 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 shutdown_connections 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 --2- 172.21.15.104:0/2968188895 >> v2:172.21.15.174:6812/17603 conn(0x7f6b34035ad0 0x7f6b34037f80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b500a0a10 0x7f6b5012d4c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b5009ff00 0x7f6b50098ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 --2- 172.21.15.104:0/2968188895 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6b5009bfd0 0x7f6b50098660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 >> 172.21.15.104:0/2968188895 conn(0x7f6b50069500 msgr2=0x7f6b500a41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 shutdown_connections 2024-09-18T14:21:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.400+0000 7f6b5830c700 1 -- 172.21.15.104:0/2968188895 wait complete. 2024-09-18T14:21:36.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.5 2024-09-18T14:21:36.419 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.3 2024-09-18T14:21:36.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.724+0000 7fcf6dffb700 1 -- 172.21.15.104:0/3314384216 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcf70017020 con 0x7fcf7c09ad20 2024-09-18T14:21:36.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.724+0000 7fcf6dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:36.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.724+0000 7fcf6dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.896+0000 7fcf8298a700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T14:21:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.896+0000 7fcf8298a700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T14:21:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.896+0000 7fcf8298a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:21:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.896+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fcf7c0979a0 con 0x7fcf7c09ad20 2024-09-18T14:21:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.896+0000 7fcf6dffb700 1 -- 172.21.15.104:0/3314384216 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf7004d060 con 0x7fcf7c09ad20 2024-09-18T14:21:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.896+0000 7fcf6dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T14:21:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.896+0000 7fcf6dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:36.905 INFO:teuthology.orchestra.run.smithi104.stdout:47244640258 2024-09-18T14:21:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 >> v2:172.21.15.174:6812/17603 conn(0x7fcf60035a80 msgr2=0x7fcf60037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3314384216 >> v2:172.21.15.174:6812/17603 conn(0x7fcf60035a80 0x7fcf60037f30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcf78007fd0 tx=0x7fcf78007b20 comp rx=0 tx=0).stop 2024-09-18T14:21:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 10 monclient: shutdown 2024-09-18T14:21:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 msgr2=0x7fcf7c06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 0x7fcf7c06ec60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000c5e0 tx=0x7fcf70004a80 comp rx=0 tx=0).stop 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 shutdown_connections 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3314384216 >> v2:172.21.15.174:6812/17603 conn(0x7fcf60035a80 0x7fcf60037f30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcf7c137e40 0x7fcf7c132c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf7c0a1900 0x7fcf7c137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 --2- 172.21.15.104:0/3314384216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c09ad20 0x7fcf7c06ec60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 >> 172.21.15.104:0/3314384216 conn(0x7fcf7c068160 msgr2=0x7fcf7c09dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 shutdown_connections 2024-09-18T14:21:36.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:36.900+0000 7fcf8298a700 1 -- 172.21.15.104:0/3314384216 wait complete. 2024-09-18T14:21:36.918 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.2 2024-09-18T14:21:37.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:21:37.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T14:21:37.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:21:37.415 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T14:21:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient: get_monmap_and_config 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient: monmap: 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:37.416021+0000 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:37.416021+0000 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:37.419 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 1 Processor -- start 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 1 -- start start 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient: init 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225409fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient(hunting): picked mon.noname-b con 0x7f225406e3f0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f225406e7c0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f225412e780 0x7f22541363e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient(hunting): picked mon.noname-a con 0x7f225412e780 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225406c900 con 0x7f225406e7c0 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225406ccc0 con 0x7f225412e780 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225406c620 con 0x7f225406e3f0 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2259da4700 10 monclient(hunting): authenticate will time out at 818.406302s 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225409fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225409fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35872/0 (socket says 172.21.15.104:35872) 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 1 -- 172.21.15.104:0/1951711775 learned_addr learned my addr 172.21.15.104:0/1951711775 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 10 monclient(hunting): get_auth_request con 0x7f225406e3f0 auth_method 0 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 1 --2- 172.21.15.104:0/1951711775 >> v2:172.21.15.104:3300/0 conn(0x7f225412e780 0x7f22541363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): get_auth_request con 0x7f225412e780 auth_method 0 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 1 -- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 msgr2=0x7f225409fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 1 --2- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225409fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 1 -- 172.21.15.104:0/1951711775 >> v1:172.21.15.174:6789/0 conn(0x7f225406e7c0 legacy=0x7f225412e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 1 monclient: found mon.noname-a 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f22595a3700 1 -- 172.21.15.104:0/1951711775 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540a1220 con 0x7f225412e780 2024-09-18T14:21:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.412+0000 7f2258da2700 1 --2- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225409fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22595a3700 1 -- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 msgr2=0x7f22541363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22595a3700 1 --2- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541363e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22540a1620 tx=0x7f224c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8928cdfe87371f4d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 1 -- 172.21.15.104:0/1951711775 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c00e070 con 0x7f225412e780 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: dump: 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.429 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 5 monclient: authenticate success, global_id 4284 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 1 -- 172.21.15.104:0/1951711775 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c00bd70 con 0x7f225412e780 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 4 set_mon_vals no callback set 2024-09-18T14:21:37.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 1 -- 172.21.15.104:0/1951711775 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c012660 con 0x7f225412e780 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f22537fe700 10 monclient: dump: 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 10 monclient: shutdown 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 -- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 msgr2=0x7f22541363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 --2- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541363e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22540a1620 tx=0x7f224c009780 comp rx=0 tx=0).stop 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 -- 172.21.15.104:0/1951711775 shutdown_connections 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 --2- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541363e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 --2- 172.21.15.104:0/1951711775 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225409fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 -- 172.21.15.104:0/1951711775 >> 172.21.15.104:0/1951711775 conn(0x7f2254067cb0 msgr2=0x7f22540a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 -- 172.21.15.104:0/1951711775 shutdown_connections 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 1 -- 172.21.15.104:0/1951711775 wait complete. 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.416+0000 7f2259da4700 10 monclient: monmap: 2024-09-18T14:21:37.432 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 Processor -- start 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 -- start start 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient: init 2024-09-18T14:21:37.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225413d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): picked mon.noname-c con 0x7f225406e3f0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225406e7c0 0x7f225413d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): picked mon.noname-b con 0x7f225406e7c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541199d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): picked mon.noname-a con 0x7f225412e780 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225409bce0 con 0x7f225412e780 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225409dcd0 con 0x7f225406e7c0 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225411d280 con 0x7f225406e3f0 2024-09-18T14:21:37.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2259da4700 10 monclient(hunting): authenticate will time out at 818.413041s 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225413d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541199d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225413d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35876/0 (socket says 172.21.15.104:35876) 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541199d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51022/0 (socket says 172.21.15.104:51022) 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 1 -- 172.21.15.104:0/101110546 learned_addr learned my addr 172.21.15.104:0/101110546 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 10 monclient(hunting): get_auth_request con 0x7f225406e3f0 auth_method 0 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): get_auth_request con 0x7f225412e780 auth_method 0 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 -- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 msgr2=0x7f225413d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225413d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 -- 172.21.15.104:0/101110546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225406e7c0 msgr2=0x7f225413d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225406e7c0 0x7f225413d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 monclient: found mon.noname-a 2024-09-18T14:21:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f22595a3700 1 -- 172.21.15.104:0/101110546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224800d990 con 0x7f225412e780 2024-09-18T14:21:37.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.420+0000 7f2258da2700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225413d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f22595a3700 1 -- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 msgr2=0x7f22541199d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f22595a3700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541199d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f224c016000 tx=0x7f224c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 1 -- 172.21.15.104:0/101110546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0263f0 con 0x7f225412e780 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: dump: 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.439 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 1 -- 172.21.15.104:0/101110546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c026570 con 0x7f225412e780 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 5 monclient: authenticate success, global_id 4287 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 10 monclient: _renew_subs 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c009a20 con 0x7f225412e780 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 1 -- 172.21.15.104:0/101110546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c026930 con 0x7f225412e780 2024-09-18T14:21:37.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 10 monclient: dump: 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 10 monclient: _renew_subs 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225411c040 con 0x7f225412e780 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 1 -- 172.21.15.104:0/101110546 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f224c02f450 con 0x7f225412e780 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 1 --2- 172.21.15.104:0/101110546 >> v2:172.21.15.174:6812/17603 conn(0x7f2238035a80 0x7f2238037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2250ff9700 1 -- 172.21.15.104:0/101110546 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f224c049f30 con 0x7f225412e780 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2252ffd700 4 set_mon_vals no callback set 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2258da2700 1 --2- 172.21.15.104:0/101110546 >> v2:172.21.15.174:6812/17603 conn(0x7f2238035a80 0x7f2238037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22340050b0 con 0x7f225412e780 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.424+0000 7f2258da2700 10 monclient: get_auth_request con 0x7f2238035a80 auth_method 0 2024-09-18T14:21:37.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.428+0000 7f2258da2700 1 --2- 172.21.15.104:0/101110546 >> v2:172.21.15.174:6812/17603 conn(0x7f2238035a80 0x7f2238037f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2248006e80 tx=0x7f224800f040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.432+0000 7f2250ff9700 1 -- 172.21.15.104:0/101110546 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f224c02f8e0 con 0x7f225412e780 2024-09-18T14:21:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.432+0000 7f2250ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.432+0000 7f2250ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient: get_monmap_and_config 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient: monmap: 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:37.449575+0000 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:37.449575+0000 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.453 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 1 Processor -- start 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 1 -- start start 2024-09-18T14:21:37.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient: init 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc8c0131d80 0x7fc8c0132150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient(hunting): picked mon.noname-c con 0x7fc8c0131d80 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8c0098010 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c0132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient(hunting): picked mon.noname-b con 0x7fc8c009bc80 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c0096640 con 0x7fc8c0098010 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c0132bd0 con 0x7fc8c0131d80 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c0096890 con 0x7fc8c009bc80 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c80d8700 10 monclient(hunting): authenticate will time out at 818.439261s 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c0132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c0132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35888/0 (socket says 172.21.15.104:35888) 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 -- 172.21.15.104:0/365802986 learned_addr learned my addr 172.21.15.104:0/365802986 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): get_auth_request con 0x7fc8c009bc80 auth_method 0 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c5e74700 1 --2- 172.21.15.104:0/365802986 >> v2:172.21.15.174:3300/0 conn(0x7fc8c0131d80 0x7fc8c0132150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c5e74700 10 monclient(hunting): get_auth_request con 0x7fc8c0131d80 auth_method 0 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c5e74700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c5e74700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c5e74700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 -- 172.21.15.104:0/365802986 >> v2:172.21.15.174:3300/0 conn(0x7fc8c0131d80 msgr2=0x7fc8c0132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 --2- 172.21.15.104:0/365802986 >> v2:172.21.15.174:3300/0 conn(0x7fc8c0131d80 0x7fc8c0132150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 -- 172.21.15.104:0/365802986 >> v1:172.21.15.174:6789/0 conn(0x7fc8c0098010 legacy=0x7fc8c012ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 monclient: found mon.noname-b 2024-09-18T14:21:37.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.444+0000 7fc8c6675700 1 -- 172.21.15.104:0/365802986 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0096b70 con 0x7fc8c009bc80 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c6675700 1 -- 172.21.15.104:0/365802986 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 msgr2=0x7fc8c0132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c6675700 1 --2- 172.21.15.104:0/365802986 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c0132690 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009f20 tx=0x7fc8b8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc33709fb9833c7d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 1 -- 172.21.15.104:0/365802986 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800e070 con 0x7fc8c009bc80 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: dump: 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.460 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 5 monclient: authenticate success, global_id 4289 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 1 -- 172.21.15.104:0/365802986 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b800bc90 con 0x7fc8c009bc80 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 4 set_mon_vals no callback set 2024-09-18T14:21:37.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 1 -- 172.21.15.104:0/365802986 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80092e0 con 0x7fc8c009bc80 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c4e72700 10 monclient: dump: 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.462 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7f8771d1c700 10 monclient: get_monmap_and_config 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7f8771d1c700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7f8771d1c700 10 monclient: monmap: 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:37.454186+0000 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:37.454186+0000 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:37.463 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:37.464 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:37.464 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:37.464 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:37.464 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 10 monclient: shutdown 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 -- 172.21.15.104:0/365802986 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 msgr2=0x7fc8c0132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/365802986 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c0132690 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009f20 tx=0x7fc8b8009840 comp rx=0 tx=0).stop 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 -- 172.21.15.104:0/365802986 shutdown_connections 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/365802986 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c0132690 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/365802986 >> v2:172.21.15.174:3300/0 conn(0x7fc8c0131d80 0x7fc8c0132150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 -- 172.21.15.104:0/365802986 >> 172.21.15.104:0/365802986 conn(0x7fc8c0067d70 msgr2=0x7fc8c00a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 -- 172.21.15.104:0/365802986 shutdown_connections 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.448+0000 7fc8c80d8700 1 -- 172.21.15.104:0/365802986 wait complete. 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient: monmap: 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 Processor -- start 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 -- start start 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient: init 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8c0098010 0x7fc8c01410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): picked mon.noname-b con 0x7fc8c0098010 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c012dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): picked mon.noname-c con 0x7fc8c009bc80 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c0131d80 0x7fc8c01178f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): picked mon.noname-a con 0x7fc8c0131d80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c0095a50 con 0x7fc8c0131d80 2024-09-18T14:21:37.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c011b0f0 con 0x7fc8c0098010 2024-09-18T14:21:37.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c01197d0 con 0x7fc8c009bc80 2024-09-18T14:21:37.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c80d8700 10 monclient(hunting): authenticate will time out at 818.446182s 2024-09-18T14:21:37.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c0131d80 0x7fc8c01178f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c0131d80 0x7fc8c01178f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51038/0 (socket says 172.21.15.104:51038) 2024-09-18T14:21:37.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 1 -- 172.21.15.104:0/2625960059 learned_addr learned my addr 172.21.15.104:0/2625960059 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 10 monclient(hunting): get_auth_request con 0x7fc8c0131d80 auth_method 0 2024-09-18T14:21:37.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 1 Processor -- start 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c012dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8c0098010 0x7fc8c01410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): get_auth_request con 0x7fc8c009bc80 auth_method 0 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 10 monclient(hunting): get_auth_request con 0x7fc8c0098010 auth_method 0 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 1 -- start start 2024-09-18T14:21:37.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient: init 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5e74700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c6675700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 1 -- 172.21.15.104:0/2625960059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8c0098010 msgr2=0x7fc8c01410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8c0098010 0x7fc8c01410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 1 -- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c0131d80 msgr2=0x7fc8c01178f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c0131d80 0x7fc8c01178f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 1 monclient: found mon.noname-c 2024-09-18T14:21:37.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7fc8c5673700 1 -- 172.21.15.104:0/2625960059 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b8009ae0 con 0x7fc8c009bc80 2024-09-18T14:21:37.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T14:21:37.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f876c097c60 0x7f876c09ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient(hunting): picked mon.noname-c con 0x7f876c097c60 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f876c09c150 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f876c09d870 0x7f876c09b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient(hunting): picked mon.noname-a con 0x7f876c09d870 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876c098030 con 0x7f876c09c150 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876c099850 con 0x7f876c09d870 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876c098fb0 con 0x7f876c097c60 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f8771d1c700 10 monclient(hunting): authenticate will time out at 818.447698s 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f876affd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f876c09c150 0x7f876c12fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47478/0 (socket says 172.21.15.104:47478) 2024-09-18T14:21:37.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f876affd700 1 -- 172.21.15.104:0/1946086591 learned_addr learned my addr 172.21.15.104:0/1946086591 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.452+0000 7f876bfff700 1 --2- 172.21.15.104:0/1946086591 >> v2:172.21.15.104:3300/0 conn(0x7f876c09d870 0x7f876c09b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 1 --2- 172.21.15.104:0/1946086591 >> v2:172.21.15.174:3300/0 conn(0x7f876c097c60 0x7f876c09ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): get_auth_request con 0x7f876c09d870 auth_method 0 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 10 monclient(hunting): get_auth_request con 0x7f876c097c60 auth_method 0 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 1 -- 172.21.15.104:0/1946086591 >> v2:172.21.15.174:3300/0 conn(0x7f876c097c60 msgr2=0x7f876c09ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 1 --2- 172.21.15.104:0/1946086591 >> v2:172.21.15.174:3300/0 conn(0x7f876c097c60 0x7f876c09ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 1 -- 172.21.15.104:0/1946086591 >> v1:172.21.15.104:6789/0 conn(0x7f876c09c150 legacy=0x7f876c12fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 1 monclient: found mon.noname-a 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 1 -- 172.21.15.104:0/1946086591 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c099290 con 0x7f876c09d870 2024-09-18T14:21:37.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876b7fe700 1 --2- 172.21.15.104:0/1946086591 >> v2:172.21.15.174:3300/0 conn(0x7f876c097c60 0x7f876c09ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:37.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7fc8c5673700 1 -- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 msgr2=0x7fc8c012dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7fc8c5673700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c012dc60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8b000c510 tx=0x7fc8b0007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 1 -- 172.21.15.104:0/1946086591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c09d870 msgr2=0x7f876c09b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.456+0000 7f876bfff700 1 --2- 172.21.15.104:0/1946086591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c09d870 0x7f876c09b2c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f876c0a22e0 tx=0x7f8764009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1133423635f03e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 1 -- 172.21.15.104:0/1946086591 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876400e070 con 0x7f876c09d870 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: dump: 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.485 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 1 -- 172.21.15.104:0/1946086591 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876400bd70 con 0x7f876c09d870 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 5 monclient: authenticate success, global_id 4293 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 1 -- 172.21.15.104:0/1946086591 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764012660 con 0x7f876c09d870 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 4 set_mon_vals no callback set 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f876a7fc700 10 monclient: dump: 2024-09-18T14:21:37.487 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 10 monclient: shutdown 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 1 -- 172.21.15.104:0/1946086591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c09d870 msgr2=0x7f876c09b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.460+0000 7f8771d1c700 1 --2- 172.21.15.104:0/1946086591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c09d870 0x7f876c09b2c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f876c0a22e0 tx=0x7f8764009780 comp rx=0 tx=0).stop 2024-09-18T14:21:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- 172.21.15.104:0/1946086591 shutdown_connections 2024-09-18T14:21:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 --2- 172.21.15.104:0/1946086591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c09d870 0x7f876c09b2c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 --2- 172.21.15.104:0/1946086591 >> v2:172.21.15.174:3300/0 conn(0x7f876c097c60 0x7f876c09ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- 172.21.15.104:0/1946086591 >> 172.21.15.104:0/1946086591 conn(0x7f876c067d70 msgr2=0x7f876c0a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- 172.21.15.104:0/1946086591 shutdown_connections 2024-09-18T14:21:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 1 -- 172.21.15.104:0/2625960059 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000ce00 con 0x7fc8c009bc80 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: dump: 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 1 -- 172.21.15.104:0/2625960059 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b00117a0 con 0x7fc8c009bc80 2024-09-18T14:21:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c80d8700 5 monclient: authenticate success, global_id 4292 2024-09-18T14:21:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c80d8700 10 monclient: _renew_subs 2024-09-18T14:21:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c80d8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c011ad80 con 0x7fc8c009bc80 2024-09-18T14:21:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- 172.21.15.104:0/1946086591 wait complete. 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 1 -- 172.21.15.104:0/2625960059 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0011b60 con 0x7fc8c009bc80 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 10 monclient: dump: 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.492 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c80d8700 10 monclient: _renew_subs 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c80d8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c0141610 con 0x7fc8c009bc80 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b7fff700 4 set_mon_vals no callback set 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient: monmap: 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 Processor -- start 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- start start 2024-09-18T14:21:37.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient: init 2024-09-18T14:21:37.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 1 -- 172.21.15.104:0/2625960059 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fc8b001c460 con 0x7fc8c009bc80 2024-09-18T14:21:37.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 1 --2- 172.21.15.104:0/2625960059 >> v2:172.21.15.174:6812/17603 conn(0x7fc8a4035ad0 0x7fc8a4037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f876c097c60 0x7f876c11c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): picked mon.noname-b con 0x7f876c097c60 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f876c09d870 0x7f876c11c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): picked mon.noname-c con 0x7f876c09d870 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c11cde0 0x7f876c1455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): picked mon.noname-a con 0x7f876c11cde0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c11dd00 con 0x7f876c11cde0 2024-09-18T14:21:37.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8b5ffb700 1 -- 172.21.15.104:0/2625960059 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fc8b00479d0 con 0x7fc8c009bc80 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c098980 con 0x7f876c097c60 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c09fd70 con 0x7f876c09d870 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f8771d1c700 10 monclient(hunting): authenticate will time out at 818.458381s 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f876c09d870 0x7f876c11c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f876c09d870 0x7f876c11c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35896/0 (socket says 172.21.15.104:35896) 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 1 -- 172.21.15.104:0/644954426 learned_addr learned my addr 172.21.15.104:0/644954426 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 10 monclient(hunting): get_auth_request con 0x7f876c09d870 auth_method 0 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c11cde0 0x7f876c1455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): get_auth_request con 0x7f876c11cde0 auth_method 0 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f876c097c60 0x7f876c11c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c5e74700 1 --2- 172.21.15.104:0/2625960059 >> v2:172.21.15.174:6812/17603 conn(0x7fc8a4035ad0 0x7fc8a4037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7fc8c5e74700 10 monclient: get_auth_request con 0x7fc8a4035ad0 auth_method 0 2024-09-18T14:21:37.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 10 monclient(hunting): get_auth_request con 0x7f876c097c60 auth_method 0 2024-09-18T14:21:37.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 1 -- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f876c09d870 msgr2=0x7f876c11c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f876c09d870 0x7f876c11c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 1 -- 172.21.15.104:0/644954426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f876c097c60 msgr2=0x7f876c11c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.501 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f876c097c60 0x7f876c11c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 1 monclient: found mon.noname-a 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.464+0000 7f876bfff700 1 -- 172.21.15.104:0/644954426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c009420 con 0x7f876c11cde0 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.468+0000 7fc8c80d8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.468+0000 7fc8c80d8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.468+0000 7fc8c80d8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.468+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a00050b0 con 0x7fc8c009bc80 2024-09-18T14:21:37.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.468+0000 7f876bfff700 1 -- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c11cde0 msgr2=0x7f876c1455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.468+0000 7f876bfff700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c11cde0 0x7f876c1455a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8764003000 tx=0x7f8764018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.468+0000 7fc8c5e74700 1 --2- 172.21.15.104:0/2625960059 >> v2:172.21.15.174:6812/17603 conn(0x7fc8a4035ad0 0x7fc8a4037f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc006fd0 tx=0x7fc8bc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7fc8b5ffb700 1 -- 172.21.15.104:0/2625960059 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc8b00108c0 con 0x7fc8c009bc80 2024-09-18T14:21:37.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7fc8b5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7fc8b5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 1 -- 172.21.15.104:0/644954426 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764009460 con 0x7f876c11cde0 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: dump: 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8771d1c700 5 monclient: authenticate success, global_id 4296 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 1 -- 172.21.15.104:0/644954426 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764021070 con 0x7f876c11cde0 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 1 -- 172.21.15.104:0/644954426 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764026430 con 0x7f876c11cde0 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8769ffb700 4 set_mon_vals no callback set 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f875b7fe700 10 monclient: dump: 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8771d1c700 10 monclient: _renew_subs 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8771d1c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8764009a20 con 0x7f876c11cde0 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8771d1c700 10 monclient: _renew_subs 2024-09-18T14:21:37.508 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8771d1c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.472+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876c12f7c0 con 0x7f876c11cde0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f875b7fe700 1 -- 172.21.15.104:0/644954426 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f8764023030 con 0x7f876c11cde0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f8771d1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f8771d1c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f8771d1c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876c064b30 con 0x7f876c11cde0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f875b7fe700 1 --2- 172.21.15.104:0/644954426 >> v2:172.21.15.174:6812/17603 conn(0x7f8750035b00 0x7f8750037fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f875b7fe700 1 -- 172.21.15.104:0/644954426 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f8764049f00 con 0x7f876c11cde0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f876b7fe700 1 --2- 172.21.15.104:0/644954426 >> v2:172.21.15.174:6812/17603 conn(0x7f8750035b00 0x7f8750037fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.476+0000 7f876b7fe700 10 monclient: get_auth_request con 0x7f8750035b00 auth_method 0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.480+0000 7f876b7fe700 1 --2- 172.21.15.104:0/644954426 >> v2:172.21.15.174:6812/17603 conn(0x7f8750035b00 0x7f8750037fb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f875c009f40 tx=0x7f875c0091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.480+0000 7f875b7fe700 1 -- 172.21.15.104:0/644954426 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f876401bbb0 con 0x7f876c11cde0 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.480+0000 7f875b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.480+0000 7f875b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient: get_monmap_and_config 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient: monmap: 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:37.519933+0000 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:37.519933+0000 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.525 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 1 Processor -- start 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 1 -- start start 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient: init 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5ec131d80 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe5ec098010 0x7fe5ec1370b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient(hunting): picked mon.noname-c con 0x7fe5ec098010 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5ec132860 0x7fe5ec139600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient(hunting): picked mon.noname-a con 0x7fe5ec132860 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec096890 con 0x7fe5ec131d80 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ec0966a0 con 0x7fe5ec132860 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ec096b70 con 0x7fe5ec098010 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f2d4d700 10 monclient(hunting): authenticate will time out at 818.509571s 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5ec132860 0x7fe5ec139600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5ec132860 0x7fe5ec139600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51070/0 (socket says 172.21.15.104:51070) 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 -- 172.21.15.104:0/391167195 learned_addr learned my addr 172.21.15.104:0/391167195 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): get_auth_request con 0x7fe5ec132860 auth_method 0 2024-09-18T14:21:37.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5ebfff700 1 --2- 172.21.15.104:0/391167195 >> v2:172.21.15.174:3300/0 conn(0x7fe5ec098010 0x7fe5ec1370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5ebfff700 10 monclient(hunting): get_auth_request con 0x7fe5ec098010 auth_method 0 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 -- 172.21.15.104:0/391167195 >> v2:172.21.15.174:3300/0 conn(0x7fe5ec098010 msgr2=0x7fe5ec1370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.528 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 --2- 172.21.15.104:0/391167195 >> v2:172.21.15.174:3300/0 conn(0x7fe5ec098010 0x7fe5ec1370b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 -- 172.21.15.104:0/391167195 >> v1:172.21.15.174:6789/0 conn(0x7fe5ec131d80 legacy=0x7fe5ec132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 monclient: found mon.noname-a 2024-09-18T14:21:37.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.529 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.516+0000 7fe5f12ea700 1 -- 172.21.15.104:0/391167195 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec09dad0 con 0x7fe5ec132860 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f12ea700 1 -- 172.21.15.104:0/391167195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec132860 msgr2=0x7fe5ec139600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f12ea700 1 --2- 172.21.15.104:0/391167195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec132860 0x7fe5ec139600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec065e40 tx=0x7fe5e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884d8cc195740bca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 1 -- 172.21.15.104:0/391167195 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e400e070 con 0x7fe5ec132860 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: dump: 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.532 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 5 monclient: authenticate success, global_id 4299 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 1 -- 172.21.15.104:0/391167195 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e400bc60 con 0x7fe5ec132860 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:37.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 1 -- 172.21.15.104:0/391167195 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e40092e0 con 0x7fe5ec132860 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 4 set_mon_vals no callback set 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5eb7fe700 10 monclient: dump: 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.534 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 10 monclient: shutdown 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/391167195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec132860 msgr2=0x7fe5ec139600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/391167195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec132860 0x7fe5ec139600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec065e40 tx=0x7fe5e4009840 comp rx=0 tx=0).stop 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/391167195 shutdown_connections 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/391167195 >> v2:172.21.15.174:3300/0 conn(0x7fe5ec098010 0x7fe5ec1370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/391167195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec132860 0x7fe5ec139600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/391167195 >> 172.21.15.104:0/391167195 conn(0x7fe5ec067d70 msgr2=0x7fe5ec0a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/391167195 shutdown_connections 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/391167195 wait complete. 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 10 monclient: monmap: 2024-09-18T14:21:37.535 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 Processor -- start 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 1 -- start start 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.520+0000 7fe5f2d4d700 10 monclient: init 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 0x7fe5ec12edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): picked mon.noname-a con 0x7fe5ec098010 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ec131d80 0x7fe5ec119910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): picked mon.noname-b con 0x7fe5ec131d80 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe5ec132860 0x7fe5ec119e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): picked mon.noname-c con 0x7fe5ec132860 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec11b4a0 con 0x7fe5ec098010 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec095a50 con 0x7fe5ec131d80 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec11d1d0 con 0x7fe5ec132860 2024-09-18T14:21:37.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient(hunting): authenticate will time out at 818.516138s 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 0x7fe5ec12edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 0x7fe5ec12edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51082/0 (socket says 172.21.15.104:51082) 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 -- 172.21.15.104:0/3824416736 learned_addr learned my addr 172.21.15.104:0/3824416736 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): get_auth_request con 0x7fe5ec098010 auth_method 0 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f12ea700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe5ec132860 0x7fe5ec119e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f12ea700 10 monclient(hunting): get_auth_request con 0x7fe5ec132860 auth_method 0 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f12ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f12ea700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f12ea700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 -- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe5ec132860 msgr2=0x7fe5ec119e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe5ec132860 0x7fe5ec119e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 -- 172.21.15.104:0/3824416736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ec131d80 msgr2=0x7fe5ec119910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5ebfff700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ec131d80 0x7fe5ec119910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ec131d80 0x7fe5ec119910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 monclient: found mon.noname-a 2024-09-18T14:21:37.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 -- 172.21.15.104:0/3824416736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4009ae0 con 0x7fe5ec098010 2024-09-18T14:21:37.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5ebfff700 10 monclient: get_auth_request con 0x7fe5ec131d80 auth_method 0 2024-09-18T14:21:37.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5ebfff700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ec131d80 0x7fe5ec119910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:21:37.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 -- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 msgr2=0x7fe5ec12edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f0ae9700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 0x7fe5ec12edc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400c5d0 tx=0x7fe5d4007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 1 -- 172.21.15.104:0/3824416736 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d40117a0 con 0x7fe5ec098010 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: dump: 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.543 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 1 -- 172.21.15.104:0/3824416736 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d4011900 con 0x7fe5ec098010 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 5 monclient: authenticate success, global_id 4302 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient: _renew_subs 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec11cef0 con 0x7fe5ec098010 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 1 -- 172.21.15.104:0/3824416736 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4011cc0 con 0x7fe5ec098010 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5ea7fc700 4 set_mon_vals no callback set 2024-09-18T14:21:37.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.524+0000 7fe5e8ff9700 10 monclient: dump: 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5f2d4d700 10 monclient: _renew_subs 2024-09-18T14:21:37.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5f2d4d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec11bf90 con 0x7fe5ec098010 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5e8ff9700 1 -- 172.21.15.104:0/3824416736 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fe5d4012630 con 0x7fe5ec098010 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5e8ff9700 1 --2- 172.21.15.104:0/3824416736 >> v2:172.21.15.174:6812/17603 conn(0x7fe5cc035a80 0x7fe5cc037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5e8ff9700 1 -- 172.21.15.104:0/3824416736 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fe5d404b120 con 0x7fe5ec098010 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5ebfff700 1 --2- 172.21.15.104:0/3824416736 >> v2:172.21.15.174:6812/17603 conn(0x7fe5cc035a80 0x7fe5cc037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5ebfff700 10 monclient: get_auth_request con 0x7fe5cc035a80 auth_method 0 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5f2d4d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5f2d4d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5f2d4d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.528+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ec064b30 con 0x7fe5ec098010 2024-09-18T14:21:37.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.532+0000 7fe5ebfff700 1 --2- 172.21.15.104:0/3824416736 >> v2:172.21.15.174:6812/17603 conn(0x7fe5cc035a80 0x7fe5cc037f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient: get_monmap_and_config 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient: monmap: 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:37.541660+0000 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:37.541660+0000 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.547 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7fe5e8ff9700 1 -- 172.21.15.104:0/3824416736 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe5d404d060 con 0x7fe5ec098010 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7fe5e8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7fe5e8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 1 Processor -- start 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 1 -- start start 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient: init 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55180a2c30 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f551806f240 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f551806f610 0x7f551813a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient(hunting): picked mon.noname-c con 0x7f551806f610 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551809ea50 con 0x7f55180a2c30 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5518099cb0 con 0x7f551806f240 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551809c7a0 con 0x7f551806f610 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f55202eb700 10 monclient(hunting): authenticate will time out at 818.531508s 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f551e087700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f55180a2c30 0x7f551809a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47480/0 (socket says 172.21.15.104:47480) 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f551e087700 1 -- 172.21.15.104:0/251018384 learned_addr learned my addr 172.21.15.104:0/251018384 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f551e888700 1 --2- 172.21.15.104:0/251018384 >> v2:172.21.15.174:3300/0 conn(0x7f551806f610 0x7f551813a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f551e888700 10 monclient(hunting): get_auth_request con 0x7f551806f610 auth_method 0 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f551e888700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f551e888700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.536+0000 7f551e888700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 1 -- 172.21.15.104:0/251018384 >> v1:172.21.15.174:6789/0 conn(0x7f551806f240 legacy=0x7f5518136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 1 -- 172.21.15.104:0/251018384 >> v1:172.21.15.104:6789/0 conn(0x7f55180a2c30 legacy=0x7f551809a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 1 monclient: found mon.noname-c 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 1 -- 172.21.15.104:0/251018384 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55180a3000 con 0x7f551806f610 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 1 -- 172.21.15.104:0/251018384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 msgr2=0x7f551813a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551e888700 1 --2- 172.21.15.104:0/251018384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 0x7f551813a5c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5510009e60 tx=0x7f5510009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b2c25e9922b282c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 1 -- 172.21.15.104:0/251018384 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551000e070 con 0x7f551806f610 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: dump: 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.555 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f55202eb700 5 monclient: authenticate success, global_id 4303 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f55202eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 1 -- 172.21.15.104:0/251018384 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5510012430 con 0x7f551806f610 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f55202eb700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f55202eb700 4 set_mon_vals no callback set 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 1 -- 172.21.15.104:0/251018384 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100127d0 con 0x7f551806f610 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:37.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.540+0000 7f551d085700 10 monclient: dump: 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient: shutdown 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- 172.21.15.104:0/251018384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 msgr2=0x7f551813a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 --2- 172.21.15.104:0/251018384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 0x7f551813a5c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5510009e60 tx=0x7f5510009780 comp rx=0 tx=0).stop 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- 172.21.15.104:0/251018384 shutdown_connections 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 --2- 172.21.15.104:0/251018384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 0x7f551813a5c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- 172.21.15.104:0/251018384 >> 172.21.15.104:0/251018384 conn(0x7f5518068f60 msgr2=0x7f55180a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- 172.21.15.104:0/251018384 shutdown_connections 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- 172.21.15.104:0/251018384 wait complete. 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient: build_initial_monmap 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient: monmap: 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:37.558 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.559 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:37.559 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:37.559 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:37.559 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 Processor -- start 2024-09-18T14:21:37.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- start start 2024-09-18T14:21:37.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient: init 2024-09-18T14:21:37.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:37.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551806f240 0x7f5518141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): picked mon.noname-a con 0x7f551806f240 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 0x7f551813a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): picked mon.noname-b con 0x7f551806f610 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 0x7f551811eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): picked mon.noname-c con 0x7f55180a2c30 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5518120790 con 0x7f551806f240 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5518098a80 con 0x7f551806f610 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55180a0a50 con 0x7f55180a2c30 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:37.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f55202eb700 10 monclient(hunting): authenticate will time out at 818.538377s 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 0x7f551811eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 0x7f551811eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35914/0 (socket says 172.21.15.104:35914) 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 -- 172.21.15.104:0/2927328062 learned_addr learned my addr 172.21.15.104:0/2927328062 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): get_auth_request con 0x7f55180a2c30 auth_method 0 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551806f240 0x7f5518141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 10 monclient(hunting): get_auth_request con 0x7f551806f240 auth_method 0 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 0x7f551813a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e087700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 10 monclient(hunting): get_auth_request con 0x7f551806f610 auth_method 0 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551d886700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-18T14:21:37.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 -- 172.21.15.104:0/2927328062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 msgr2=0x7f551813a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 0x7f551813a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 -- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551806f240 msgr2=0x7f5518141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551806f240 0x7f5518141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient: _finish_hunting 0 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 monclient: found mon.noname-c 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.544+0000 7f551e888700 1 -- 172.21.15.104:0/2927328062 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5510009a20 con 0x7f55180a2c30 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f551e888700 1 -- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 msgr2=0x7f551811eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f551e888700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 0x7f551811eb70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5510012950 tx=0x7f5510018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 1 -- 172.21.15.104:0/2927328062 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5510026430 con 0x7f55180a2c30 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: dump: 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:37.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f55202eb700 5 monclient: authenticate success, global_id 4301 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 1 -- 172.21.15.104:0/2927328062 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5510021070 con 0x7f55180a2c30 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f55202eb700 10 monclient: _renew_subs 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f55202eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55181221c0 con 0x7f55180a2c30 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 1 -- 172.21.15.104:0/2927328062 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100267e0 con 0x7f55180a2c30 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 10 monclient: dump: 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:37.568 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f55202eb700 10 monclient: _renew_subs 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f55202eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55181415b0 con 0x7f55180a2c30 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550ffff700 4 set_mon_vals no callback set 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 1 -- 172.21.15.104:0/2927328062 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f5510023030 con 0x7f55180a2c30 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 1 --2- 172.21.15.104:0/2927328062 >> v2:172.21.15.174:6812/17603 conn(0x7f5504035c10 0x7f55040380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f550e7fc700 1 -- 172.21.15.104:0/2927328062 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f551004a0e0 con 0x7f55180a2c30 2024-09-18T14:21:37.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f551e087700 1 --2- 172.21.15.104:0/2927328062 >> v2:172.21.15.174:6812/17603 conn(0x7f5504035c10 0x7f55040380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:37.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.548+0000 7f551e087700 10 monclient: get_auth_request con 0x7f5504035c10 auth_method 0 2024-09-18T14:21:37.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.552+0000 7f551e087700 1 --2- 172.21.15.104:0/2927328062 >> v2:172.21.15.174:6812/17603 conn(0x7f5504035c10 0x7f55040380c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5508007880 tx=0x7f5508007f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:37.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.556+0000 7f55202eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.556+0000 7f55202eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.556+0000 7f55202eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.556+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55000050b0 con 0x7f55180a2c30 2024-09-18T14:21:37.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.564+0000 7f550e7fc700 1 -- 172.21.15.104:0/2927328062 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f551002f440 con 0x7f55180a2c30 2024-09-18T14:21:37.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.564+0000 7f550e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:37.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.564+0000 7f550e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:37.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T14:21:37.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T14:21:37.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc8a0005360 con 0x7fc8c009bc80 2024-09-18T14:21:37.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8b5ffb700 1 -- 172.21.15.104:0/2625960059 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc8b00102e0 con 0x7fc8c009bc80 2024-09-18T14:21:37.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8b5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T14:21:37.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8b5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:37.653 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:37.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 >> v2:172.21.15.174:6812/17603 conn(0x7fc8a4035ad0 msgr2=0x7fc8a4037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/2625960059 >> v2:172.21.15.174:6812/17603 conn(0x7fc8a4035ad0 0x7fc8a4037f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc006fd0 tx=0x7fc8bc008040 comp rx=0 tx=0).stop 2024-09-18T14:21:37.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 10 monclient: shutdown 2024-09-18T14:21:37.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 msgr2=0x7fc8c012dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c012dc60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8b000c510 tx=0x7fc8b0007ea0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 shutdown_connections 2024-09-18T14:21:37.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/2625960059 >> v2:172.21.15.174:6812/17603 conn(0x7fc8a4035ad0 0x7fc8a4037f80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c0131d80 0x7fc8c01178f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc8c009bc80 0x7fc8c012dc60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 --2- 172.21.15.104:0/2625960059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8c0098010 0x7fc8c01410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.644+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 >> 172.21.15.104:0/2625960059 conn(0x7fc8c0067d70 msgr2=0x7fc8c0134d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.648+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 shutdown_connections 2024-09-18T14:21:37.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.648+0000 7fc8c80d8700 1 -- 172.21.15.104:0/2625960059 wait complete. 2024-09-18T14:21:37.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.4 2024-09-18T14:21:37.668 DEBUG:teuthology.parallel:result is None 2024-09-18T14:21:37.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.700+0000 7f8771d1c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T14:21:37.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.700+0000 7f8771d1c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T14:21:37.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.700+0000 7f8771d1c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.700+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f876c11f940 con 0x7f876c11cde0 2024-09-18T14:21:37.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.700+0000 7f875b7fe700 1 -- 172.21.15.104:0/644954426 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f876401b810 con 0x7f876c11cde0 2024-09-18T14:21:37.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.700+0000 7f875b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T14:21:37.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.700+0000 7f875b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:37.710 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:37.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 >> v2:172.21.15.174:6812/17603 conn(0x7f8750035b00 msgr2=0x7f8750037fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 --2- 172.21.15.104:0/644954426 >> v2:172.21.15.174:6812/17603 conn(0x7f8750035b00 0x7f8750037fb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f875c009f40 tx=0x7f875c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 10 monclient: shutdown 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c11cde0 msgr2=0x7f876c1455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c11cde0 0x7f876c1455a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8764003000 tx=0x7f8764018000 comp rx=0 tx=0).stop 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 shutdown_connections 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 --2- 172.21.15.104:0/644954426 >> v2:172.21.15.174:6812/17603 conn(0x7f8750035b00 0x7f8750037fb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c11cde0 0x7f876c1455a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f876c09d870 0x7f876c11c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 --2- 172.21.15.104:0/644954426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f876c097c60 0x7f876c11c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 >> 172.21.15.104:0/644954426 conn(0x7f876c067d70 msgr2=0x7f876c06e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 shutdown_connections 2024-09-18T14:21:37.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.704+0000 7f8771d1c700 1 -- 172.21.15.104:0/644954426 wait complete. 2024-09-18T14:21:37.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-09-18T14:21:37.722 DEBUG:teuthology.parallel:result is None 2024-09-18T14:21:37.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.716+0000 7f2259da4700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T14:21:37.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.716+0000 7f2259da4700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T14:21:37.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.716+0000 7f2259da4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.716+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2234005360 con 0x7f225412e780 2024-09-18T14:21:37.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.716+0000 7f2250ff9700 1 -- 172.21.15.104:0/101110546 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f224c02e820 con 0x7f225412e780 2024-09-18T14:21:37.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.716+0000 7f2250ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T14:21:37.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.716+0000 7f2250ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:37.726 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 >> v2:172.21.15.174:6812/17603 conn(0x7f2238035a80 msgr2=0x7f2238037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 --2- 172.21.15.104:0/101110546 >> v2:172.21.15.174:6812/17603 conn(0x7f2238035a80 0x7f2238037f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2248006e80 tx=0x7f224800f040 comp rx=0 tx=0).stop 2024-09-18T14:21:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 10 monclient: shutdown 2024-09-18T14:21:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 msgr2=0x7f22541199d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541199d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f224c016000 tx=0x7f224c019000 comp rx=0 tx=0).stop 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 shutdown_connections 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 --2- 172.21.15.104:0/101110546 >> v2:172.21.15.174:6812/17603 conn(0x7f2238035a80 0x7f2238037f30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225412e780 0x7f22541199d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f225406e7c0 0x7f225413d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 --2- 172.21.15.104:0/101110546 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f225406e3f0 0x7f225413d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 >> 172.21.15.104:0/101110546 conn(0x7f2254067cb0 msgr2=0x7f225409e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 shutdown_connections 2024-09-18T14:21:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.720+0000 7f2259da4700 1 -- 172.21.15.104:0/101110546 wait complete. 2024-09-18T14:21:37.737 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-18T14:21:37.737 DEBUG:teuthology.parallel:result is None 2024-09-18T14:21:37.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.752+0000 7f55202eb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T14:21:37.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.752+0000 7f55202eb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T14:21:37.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.752+0000 7f55202eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:37.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.752+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5500005360 con 0x7f55180a2c30 2024-09-18T14:21:37.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f550e7fc700 1 -- 172.21.15.104:0/2927328062 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5510037090 con 0x7f55180a2c30 2024-09-18T14:21:37.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f550e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T14:21:37.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f550e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:37.763 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 >> v2:172.21.15.174:6812/17603 conn(0x7f5504035c10 msgr2=0x7f55040380c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 --2- 172.21.15.104:0/2927328062 >> v2:172.21.15.174:6812/17603 conn(0x7f5504035c10 0x7f55040380c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5508007880 tx=0x7f5508007f90 comp rx=0 tx=0).stop 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 10 monclient: shutdown 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 msgr2=0x7f551811eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 0x7f551811eb70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5510012950 tx=0x7f5510018000 comp rx=0 tx=0).stop 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 shutdown_connections 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 --2- 172.21.15.104:0/2927328062 >> v2:172.21.15.174:6812/17603 conn(0x7f5504035c10 0x7f55040380c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55180a2c30 0x7f551811eb70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f551806f610 0x7f551813a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 --2- 172.21.15.104:0/2927328062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551806f240 0x7f5518141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 >> 172.21.15.104:0/2927328062 conn(0x7f5518068f60 msgr2=0x7f551809b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 shutdown_connections 2024-09-18T14:21:37.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.756+0000 7f55202eb700 1 -- 172.21.15.104:0/2927328062 wait complete. 2024-09-18T14:21:37.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.764+0000 7fe5f2d4d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T14:21:37.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.764+0000 7fe5f2d4d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T14:21:37.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.764+0000 7fe5f2d4d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:37.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.764+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe5ec068ce0 con 0x7fe5ec098010 2024-09-18T14:21:37.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.764+0000 7fe5e8ff9700 1 -- 172.21.15.104:0/3824416736 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe5d4010680 con 0x7fe5ec098010 2024-09-18T14:21:37.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.764+0000 7fe5e8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T14:21:37.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.764+0000 7fe5e8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:37.773 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:37.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 >> v2:172.21.15.174:6812/17603 conn(0x7fe5cc035a80 msgr2=0x7fe5cc037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/3824416736 >> v2:172.21.15.174:6812/17603 conn(0x7fe5cc035a80 0x7fe5cc037f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc008040 comp rx=0 tx=0).stop 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 10 monclient: shutdown 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 msgr2=0x7fe5ec12edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 0x7fe5ec12edc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400c5d0 tx=0x7fe5d4007f60 comp rx=0 tx=0).stop 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 shutdown_connections 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/3824416736 >> v2:172.21.15.174:6812/17603 conn(0x7fe5cc035a80 0x7fe5cc037f30 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc008040 comp rx=0 tx=0).stop 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe5ec132860 0x7fe5ec119e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ec131d80 0x7fe5ec119910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 --2- 172.21.15.104:0/3824416736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec098010 0x7fe5ec12edc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 >> 172.21.15.104:0/3824416736 conn(0x7fe5ec067d70 msgr2=0x7fe5ec131590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 shutdown_connections 2024-09-18T14:21:37.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:37.768+0000 7fe5f2d4d700 1 -- 172.21.15.104:0/3824416736 wait complete. 2024-09-18T14:21:37.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-09-18T14:21:37.777 DEBUG:teuthology.parallel:result is None 2024-09-18T14:21:37.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-18T14:21:37.786 DEBUG:teuthology.parallel:result is None 2024-09-18T14:21:37.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:21:38.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient: get_monmap_and_config 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient: monmap: 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:38.025912+0000 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:38.025912+0000 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:38.030 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:38.030 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:38.030 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 1 Processor -- start 2024-09-18T14:21:38.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 1 -- start start 2024-09-18T14:21:38.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient: init 2024-09-18T14:21:38.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T14:21:38.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc7f009e7b0 0x7fc7f009fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient(hunting): picked mon.noname-c con 0x7fc7f009e7b0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7f006fa10 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc7f0137780 0x7fc7f0139b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient(hunting): picked mon.noname-a con 0x7fc7f0137780 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f00a1e80 con 0x7fc7f006fa10 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f009dd70 con 0x7fc7f0137780 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f009cae0 con 0x7fc7f009e7b0 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7f71de700 10 monclient(hunting): authenticate will time out at 819.015746s 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7effff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc7f006fa10 0x7fc7f00a0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47486/0 (socket says 172.21.15.104:47486) 2024-09-18T14:21:38.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.020+0000 7fc7effff700 1 -- 172.21.15.104:0/2023714448 learned_addr learned my addr 172.21.15.104:0/2023714448 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 --2- 172.21.15.104:0/2023714448 >> v2:172.21.15.104:3300/0 conn(0x7fc7f0137780 0x7fc7f0139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): get_auth_request con 0x7fc7f0137780 auth_method 0 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 1 --2- 172.21.15.104:0/2023714448 >> v2:172.21.15.174:3300/0 conn(0x7fc7f009e7b0 0x7fc7f009fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 10 monclient(hunting): get_auth_request con 0x7fc7f009e7b0 auth_method 0 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f4f7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 -- 172.21.15.104:0/2023714448 >> v2:172.21.15.174:3300/0 conn(0x7fc7f009e7b0 msgr2=0x7fc7f009fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 --2- 172.21.15.104:0/2023714448 >> v2:172.21.15.174:3300/0 conn(0x7fc7f009e7b0 0x7fc7f009fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 -- 172.21.15.104:0/2023714448 >> v1:172.21.15.104:6789/0 conn(0x7fc7f006fa10 legacy=0x7fc7f00a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 monclient: found mon.noname-a 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 -- 172.21.15.104:0/2023714448 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f009eb80 con 0x7fc7f0137780 2024-09-18T14:21:38.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 -- 172.21.15.104:0/2023714448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f0137780 msgr2=0x7fc7f0139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f577b700 1 --2- 172.21.15.104:0/2023714448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f0137780 0x7fc7f0139b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0072340 tx=0x7fc7e8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bb69f3cf64e4e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 1 -- 172.21.15.104:0/2023714448 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e800e070 con 0x7fc7f0137780 2024-09-18T14:21:38.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:38.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: dump: 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 1 -- 172.21.15.104:0/2023714448 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e800bd70 con 0x7fc7f0137780 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f71de700 5 monclient: authenticate success, global_id 4308 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f71de700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f71de700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7f71de700 4 set_mon_vals no callback set 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 1 -- 172.21.15.104:0/2023714448 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8012660 con 0x7fc7f0137780 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.024+0000 7fc7ef7fe700 10 monclient: dump: 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.039 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 10 monclient: shutdown 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 -- 172.21.15.104:0/2023714448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f0137780 msgr2=0x7fc7f0139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 --2- 172.21.15.104:0/2023714448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f0137780 0x7fc7f0139b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0072340 tx=0x7fc7e8009780 comp rx=0 tx=0).stop 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 -- 172.21.15.104:0/2023714448 shutdown_connections 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 --2- 172.21.15.104:0/2023714448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f0137780 0x7fc7f0139b60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 --2- 172.21.15.104:0/2023714448 >> v2:172.21.15.174:3300/0 conn(0x7fc7f009e7b0 0x7fc7f009fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 -- 172.21.15.104:0/2023714448 >> 172.21.15.104:0/2023714448 conn(0x7fc7f0068120 msgr2=0x7fc7f00a5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 -- 172.21.15.104:0/2023714448 shutdown_connections 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 -- 172.21.15.104:0/2023714448 wait complete. 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 10 monclient: monmap: 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.041 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:38.042 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:38.042 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:38.042 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 Processor -- start 2024-09-18T14:21:38.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 1 -- start start 2024-09-18T14:21:38.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.028+0000 7fc7f71de700 10 monclient: init 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f006fa10 0x7fc7f0136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): picked mon.noname-b con 0x7fc7f006fa10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 0x7fc7f0135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): picked mon.noname-a con 0x7fc7f009e7b0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc7f0137780 0x7fc7f012eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): picked mon.noname-c con 0x7fc7f0137780 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0130760 con 0x7fc7f009e7b0 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0135760 con 0x7fc7f006fa10 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f009cd10 con 0x7fc7f0137780 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient(hunting): authenticate will time out at 819.024497s 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 0x7fc7f0135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 0x7fc7f0135220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51110/0 (socket says 172.21.15.104:51110) 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc7f0137780 0x7fc7f012eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 -- 172.21.15.104:0/1230885 learned_addr learned my addr 172.21.15.104:0/1230885 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): get_auth_request con 0x7fc7f009e7b0 auth_method 0 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 10 monclient(hunting): get_auth_request con 0x7fc7f0137780 auth_method 0 2024-09-18T14:21:38.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f006fa10 0x7fc7f0136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 10 monclient(hunting): get_auth_request con 0x7fc7f006fa10 auth_method 0 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f577b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f4f7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 -- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc7f0137780 msgr2=0x7fc7f012eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc7f0137780 0x7fc7f012eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 -- 172.21.15.104:0/1230885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f006fa10 msgr2=0x7fc7f0136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f006fa10 0x7fc7f0136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 monclient: found mon.noname-a 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 -- 172.21.15.104:0/1230885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e8009a20 con 0x7fc7f009e7b0 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 -- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 msgr2=0x7fc7f0135220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7effff700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 0x7fc7f0135220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000c5e0 tx=0x7fc7e0004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 1 -- 172.21.15.104:0/1230885 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0005d40 con 0x7fc7f009e7b0 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 10 monclient: dump: 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 5 monclient: authenticate success, global_id 4311 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 1 -- 172.21.15.104:0/1230885 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e00070d0 con 0x7fc7f009e7b0 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient: _renew_subs 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f01324f0 con 0x7fc7f009e7b0 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.032+0000 7fc7ecff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7ecff9700 1 -- 172.21.15.104:0/1230885 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0007490 con 0x7fc7f009e7b0 2024-09-18T14:21:38.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7ecff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7ecff9700 10 monclient: dump: 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f71de700 10 monclient: _renew_subs 2024-09-18T14:21:38.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7eeffd700 4 set_mon_vals no callback set 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f71de700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f01312b0 con 0x7fc7f009e7b0 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f71de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f71de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f71de700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f0064b30 con 0x7fc7f009e7b0 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7ecff9700 1 -- 172.21.15.104:0/1230885 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fc7e001b3b0 con 0x7fc7f009e7b0 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7ecff9700 1 --2- 172.21.15.104:0/1230885 >> v2:172.21.15.174:6812/17603 conn(0x7fc7dc035a80 0x7fc7dc037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7ecff9700 1 -- 172.21.15.104:0/1230885 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fc7e0047440 con 0x7fc7f009e7b0 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f4f7a700 1 --2- 172.21.15.104:0/1230885 >> v2:172.21.15.174:6812/17603 conn(0x7fc7dc035a80 0x7fc7dc037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.036+0000 7fc7f4f7a700 10 monclient: get_auth_request con 0x7fc7dc035a80 auth_method 0 2024-09-18T14:21:38.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.040+0000 7fc7f4f7a700 1 --2- 172.21.15.104:0/1230885 >> v2:172.21.15.174:6812/17603 conn(0x7fc7dc035a80 0x7fc7dc037f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009db0 tx=0x7fc7e40091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.040+0000 7fc7ecff9700 1 -- 172.21.15.104:0/1230885 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc7e00198c0 con 0x7fc7f009e7b0 2024-09-18T14:21:38.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.040+0000 7fc7ecff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.040+0000 7fc7ecff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.200+0000 7fc7f71de700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T14:21:38.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.200+0000 7fc7f71de700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T14:21:38.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.200+0000 7fc7f71de700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.200+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc7f0132710 con 0x7fc7f009e7b0 2024-09-18T14:21:38.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.200+0000 7fc7ecff9700 1 -- 172.21.15.104:0/1230885 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc7e0017070 con 0x7fc7f009e7b0 2024-09-18T14:21:38.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.200+0000 7fc7ecff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T14:21:38.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.200+0000 7fc7ecff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:38.209 INFO:teuthology.orchestra.run.smithi104.stdout:47244640260 2024-09-18T14:21:38.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 >> v2:172.21.15.174:6812/17603 conn(0x7fc7dc035a80 msgr2=0x7fc7dc037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 --2- 172.21.15.104:0/1230885 >> v2:172.21.15.174:6812/17603 conn(0x7fc7dc035a80 0x7fc7dc037f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009db0 tx=0x7fc7e40091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 10 monclient: shutdown 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 msgr2=0x7fc7f0135220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 0x7fc7f0135220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000c5e0 tx=0x7fc7e0004a80 comp rx=0 tx=0).stop 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 shutdown_connections 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 --2- 172.21.15.104:0/1230885 >> v2:172.21.15.174:6812/17603 conn(0x7fc7dc035a80 0x7fc7dc037f30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc7f0137780 0x7fc7f012eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7f009e7b0 0x7fc7f0135220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 --2- 172.21.15.104:0/1230885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f006fa10 0x7fc7f0136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 >> 172.21.15.104:0/1230885 conn(0x7fc7f0068120 msgr2=0x7fc7f0136f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:38.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 shutdown_connections 2024-09-18T14:21:38.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.204+0000 7fc7f71de700 1 -- 172.21.15.104:0/1230885 wait complete. 2024-09-18T14:21:38.223 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-18T14:21:38.223 DEBUG:teuthology.parallel:result is None 2024-09-18T14:21:38.223 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T14:21:38.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:38.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient: get_monmap_and_config 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient: monmap: 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:38.321826+0000 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:38.321826+0000 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:38.326 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:38.327 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:38.327 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:38.327 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:38.327 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 1 Processor -- start 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 1 -- start start 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient: init 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbda8063b20 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbda8130480 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbda809c330 0x7fbda8138340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient(hunting): picked mon.noname-c con 0x7fbda809c330 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:38.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda809d2c0 con 0x7fbda8063b20 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda8063ef0 con 0x7fbda8130480 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda806f190 con 0x7fbda809c330 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbdace34700 10 monclient(hunting): authenticate will time out at 819.311414s 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbda77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbda8063b20 0x7fbda812fd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47500/0 (socket says 172.21.15.104:47500) 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbda77fe700 1 -- 172.21.15.104:0/3440851565 learned_addr learned my addr 172.21.15.104:0/3440851565 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbda7fff700 1 --2- 172.21.15.104:0/3440851565 >> v2:172.21.15.174:3300/0 conn(0x7fbda809c330 0x7fbda8138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbda7fff700 10 monclient(hunting): get_auth_request con 0x7fbda809c330 auth_method 0 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbda7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbda7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.316+0000 7fbda7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 1 -- 172.21.15.104:0/3440851565 >> v1:172.21.15.174:6789/0 conn(0x7fbda8130480 legacy=0x7fbda8134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 1 -- 172.21.15.104:0/3440851565 >> v1:172.21.15.104:6789/0 conn(0x7fbda8063b20 legacy=0x7fbda812fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 1 monclient: found mon.noname-c 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 1 -- 172.21.15.104:0/3440851565 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda812f920 con 0x7fbda809c330 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 1 -- 172.21.15.104:0/3440851565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 msgr2=0x7fbda8138340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda7fff700 1 --2- 172.21.15.104:0/3440851565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8138340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbda8065630 tx=0x7fbda00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d21422f987bb3099 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 1 -- 172.21.15.104:0/3440851565 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000b9a0 con 0x7fbda809c330 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 10 monclient: dump: 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 1 -- 172.21.15.104:0/3440851565 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda000bb00 con 0x7fbda809c330 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 1 -- 172.21.15.104:0/3440851565 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000bea0 con 0x7fbda809c330 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbdace34700 5 monclient: authenticate success, global_id 4315 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbdace34700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbdace34700 4 set_mon_vals no callback set 2024-09-18T14:21:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.320+0000 7fbda67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda67fc700 10 monclient: dump: 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient: shutdown 2024-09-18T14:21:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- 172.21.15.104:0/3440851565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 msgr2=0x7fbda8138340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 --2- 172.21.15.104:0/3440851565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8138340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbda8065630 tx=0x7fbda00091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- 172.21.15.104:0/3440851565 shutdown_connections 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 --2- 172.21.15.104:0/3440851565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8138340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- 172.21.15.104:0/3440851565 >> 172.21.15.104:0/3440851565 conn(0x7fbda8068160 msgr2=0x7fbda80a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- 172.21.15.104:0/3440851565 shutdown_connections 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- 172.21.15.104:0/3440851565 wait complete. 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient: monmap: 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 Processor -- start 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- start start 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient: init 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 0x7fbda8118640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): picked mon.noname-a con 0x7fbda8063b20 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8118b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): picked mon.noname-b con 0x7fbda809c330 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbda8130480 0x7fbda81190c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): picked mon.noname-c con 0x7fbda8130480 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda809c9a0 con 0x7fbda8063b20 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda811bfc0 con 0x7fbda809c330 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda812e8f0 con 0x7fbda8130480 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbdace34700 10 monclient(hunting): authenticate will time out at 819.318775s 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 0x7fbda8118640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 0x7fbda8118640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51124/0 (socket says 172.21.15.104:51124) 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 -- 172.21.15.104:0/279802848 learned_addr learned my addr 172.21.15.104:0/279802848 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): get_auth_request con 0x7fbda8063b20 auth_method 0 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbda8130480 0x7fbda81190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 10 monclient(hunting): get_auth_request con 0x7fbda8130480 auth_method 0 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda6ffd700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8118b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda6ffd700 10 monclient(hunting): get_auth_request con 0x7fbda809c330 auth_method 0 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 -- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbda8130480 msgr2=0x7fbda81190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbda8130480 0x7fbda81190c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 -- 172.21.15.104:0/279802848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 msgr2=0x7fbda8118b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8118b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 monclient: found mon.noname-a 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda77fe700 1 -- 172.21.15.104:0/279802848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0009420 con 0x7fbda8063b20 2024-09-18T14:21:38.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.324+0000 7fbda6ffd700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8118b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:38.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbda77fe700 1 -- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 msgr2=0x7fbda8118640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbda77fe700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 0x7fbda8118640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000b5e0 tx=0x7fbd90009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 1 -- 172.21.15.104:0/279802848 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90011840 con 0x7fbda8063b20 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: dump: 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.346 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 1 -- 172.21.15.104:0/279802848 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd900119a0 con 0x7fbda8063b20 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 5 monclient: authenticate success, global_id 4314 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 10 monclient: _renew_subs 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda811a190 con 0x7fbda8063b20 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 1 -- 172.21.15.104:0/279802848 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90011d60 con 0x7fbda8063b20 2024-09-18T14:21:38.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 10 monclient: dump: 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 10 monclient: _renew_subs 2024-09-18T14:21:38.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbda5ffb700 4 set_mon_vals no callback set 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda811c9a0 con 0x7fbda8063b20 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbdace34700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.328+0000 7fbd9f7fe700 1 -- 172.21.15.104:0/279802848 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fbd90010590 con 0x7fbda8063b20 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbdace34700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbdace34700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbd9f7fe700 1 --2- 172.21.15.104:0/279802848 >> v2:172.21.15.174:6812/17603 conn(0x7fbd88035a80 0x7fbd88037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda8119b40 con 0x7fbda8063b20 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbd9f7fe700 1 -- 172.21.15.104:0/279802848 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fbd90047f60 con 0x7fbda8063b20 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbda6ffd700 1 --2- 172.21.15.104:0/279802848 >> v2:172.21.15.174:6812/17603 conn(0x7fbd88035a80 0x7fbd88037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbda6ffd700 10 monclient: get_auth_request con 0x7fbd88035a80 auth_method 0 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.332+0000 7fbda6ffd700 1 --2- 172.21.15.104:0/279802848 >> v2:172.21.15.174:6812/17603 conn(0x7fbd88035a80 0x7fbd88037f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbda8119e20 tx=0x7fbd98009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.336+0000 7fbd9f7fe700 1 -- 172.21.15.104:0/279802848 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbd90015030 con 0x7fbda8063b20 2024-09-18T14:21:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.336+0000 7fbd9f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.336+0000 7fbd9f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:38.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.496+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbda811c100 con 0x7fbd88035a80 2024-09-18T14:21:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbd9f7fe700 1 -- 172.21.15.104:0/279802848 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7fbd98002c20 con 0x7fbd88035a80 2024-09-18T14:21:38.511 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 >> v2:172.21.15.174:6812/17603 conn(0x7fbd88035a80 msgr2=0x7fbd88037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 --2- 172.21.15.104:0/279802848 >> v2:172.21.15.174:6812/17603 conn(0x7fbd88035a80 0x7fbd88037f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbda8119e20 tx=0x7fbd98009040 comp rx=0 tx=0).stop 2024-09-18T14:21:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 10 monclient: shutdown 2024-09-18T14:21:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 msgr2=0x7fbda8118640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 0x7fbda8118640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000b5e0 tx=0x7fbd90009120 comp rx=0 tx=0).stop 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 shutdown_connections 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 --2- 172.21.15.104:0/279802848 >> v2:172.21.15.174:6812/17603 conn(0x7fbd88035a80 0x7fbd88037f30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbda8130480 0x7fbda81190c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbda809c330 0x7fbda8118b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 --2- 172.21.15.104:0/279802848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbda8063b20 0x7fbda8118640 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 >> 172.21.15.104:0/279802848 conn(0x7fbda8068160 msgr2=0x7fbda806d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 shutdown_connections 2024-09-18T14:21:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.504+0000 7fbdace34700 1 -- 172.21.15.104:0/279802848 wait complete. 2024-09-18T14:21:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:21:38.532 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-18T14:21:38.405351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4778452,"kb_used_data":557418280,"kb_used_omap":444,"kb_used_meta":0,"kb_avail":557417828,"statfs":{"total":575688990720,"available":570795855872,"internally_reserved":462848,"allocated":570796318720,"data_stored":570796318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":455649,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.130687"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135537+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:21:35.135537+0000","last_peered":"2024-09-18T14:21:35.135537+0000","last_clean":"2024-09-18T14:21:35.135537+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:21:35.135537+0000","last_undegraded":"2024-09-18T14:21:35.135537+0000","last_fullsized":"2024-09-18T14:21:35.135537+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295699+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:21:34.295699+0000","last_peered":"2024-09-18T14:21:34.295699+0000","last_clean":"2024-09-18T14:21:34.295699+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:21:34.295699+0000","last_undegraded":"2024-09-18T14:21:34.295699+0000","last_fullsized":"2024-09-18T14:21:34.295699+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135401+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:21:35.135401+0000","last_peered":"2024-09-18T14:21:35.135401+0000","last_clean":"2024-09-18T14:21:35.135401+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:21:35.135401+0000","last_undegraded":"2024-09-18T14:21:35.135401+0000","last_fullsized":"2024-09-18T14:21:35.135401+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295840+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:21:34.295840+0000","last_peered":"2024-09-18T14:21:34.295840+0000","last_clean":"2024-09-18T14:21:34.295840+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:21:34.295840+0000","last_undegraded":"2024-09-18T14:21:34.295840+0000","last_fullsized":"2024-09-18T14:21:34.295840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296315+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:21:34.296315+0000","last_peered":"2024-09-18T14:21:34.296315+0000","last_clean":"2024-09-18T14:21:34.296315+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:21:34.296315+0000","last_undegraded":"2024-09-18T14:21:34.296315+0000","last_fullsized":"2024-09-18T14:21:34.296315+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.349394+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:21:34.349394+0000","last_peered":"2024-09-18T14:21:34.349394+0000","last_clean":"2024-09-18T14:21:34.349394+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:21:34.349394+0000","last_undegraded":"2024-09-18T14:21:34.349394+0000","last_fullsized":"2024-09-18T14:21:34.349394+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135845+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:21:35.135845+0000","last_peered":"2024-09-18T14:21:35.135845+0000","last_clean":"2024-09-18T14:21:35.135845+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:21:35.135845+0000","last_undegraded":"2024-09-18T14:21:35.135845+0000","last_fullsized":"2024-09-18T14:21:35.135845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296035+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:21:34.296035+0000","last_peered":"2024-09-18T14:21:34.296035+0000","last_clean":"2024-09-18T14:21:34.296035+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:21:34.296035+0000","last_undegraded":"2024-09-18T14:21:34.296035+0000","last_fullsized":"2024-09-18T14:21:34.296035+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T14:21:33.278313+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:21:33.278313+0000","last_peered":"2024-09-18T14:21:33.278313+0000","last_clean":"2024-09-18T14:21:33.278313+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:21:33.278313+0000","last_undegraded":"2024-09-18T14:21:33.278313+0000","last_fullsized":"2024-09-18T14:21:33.278313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796012,"kb_used_data":92903436,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903368,"statfs":{"total":95948165120,"available":95133048832,"internally_reserved":69632,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58529,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795964,"kb_used_data":92903460,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903416,"statfs":{"total":95948165120,"available":95133097984,"internally_reserved":45056,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48713,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797228,"kb_used_data":92902244,"kb_used_omap":110,"kb_used_meta":0,"kb_avail":92902152,"statfs":{"total":95948165120,"available":95131803648,"internally_reserved":94208,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":113606,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795980,"kb_used_data":92903460,"kb_used_omap":54,"kb_used_meta":0,"kb_avail":92903400,"statfs":{"total":95948165120,"available":95133081600,"internally_reserved":61440,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55928,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796068,"kb_used_data":92903436,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":92903312,"statfs":{"total":95948165120,"available":95132991488,"internally_reserved":126976,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77686,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797200,"kb_used_data":92902244,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92902180,"statfs":{"total":95948165120,"available":95131832320,"internally_reserved":65536,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101187,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]}]}],"pool_statfs":[]}} 2024-09-18T14:21:38.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:38.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient: get_monmap_and_config 2024-09-18T14:21:38.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient: monmap: 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:38.628862+0000 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:38.628862+0000 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:38.632 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 1 Processor -- start 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 1 -- start start 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient: init 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f664806f440 0x7f664806b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient(hunting): picked mon.noname-c con 0x7f664806f440 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6648131b70 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:38.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648132680 0x7f6648138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient(hunting): picked mon.noname-b con 0x7f6648132680 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664806ad30 con 0x7f6648131b70 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664806b010 con 0x7f664806f440 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664806bc50 con 0x7f6648132680 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664df1c700 10 monclient(hunting): authenticate will time out at 819.618777s 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f6647fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6648131b70 0x7f6648131f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47508/0 (socket says 172.21.15.104:47508) 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f6647fff700 1 -- 172.21.15.104:0/3082021279 learned_addr learned my addr 172.21.15.104:0/3082021279 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 1 --2- 172.21.15.104:0/3082021279 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648132680 0x7f6648138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): get_auth_request con 0x7f6648132680 auth_method 0 2024-09-18T14:21:38.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 1 --2- 172.21.15.104:0/3082021279 >> v2:172.21.15.174:3300/0 conn(0x7f664806f440 0x7f664806b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 10 monclient(hunting): get_auth_request con 0x7f664806f440 auth_method 0 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664cf1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 1 -- 172.21.15.104:0/3082021279 >> v2:172.21.15.174:3300/0 conn(0x7f664806f440 msgr2=0x7f664806b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 1 --2- 172.21.15.104:0/3082021279 >> v2:172.21.15.174:3300/0 conn(0x7f664806f440 0x7f664806b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 1 -- 172.21.15.104:0/3082021279 >> v1:172.21.15.104:6789/0 conn(0x7f6648131b70 legacy=0x7f6648131f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 1 monclient: found mon.noname-b 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:38.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.624+0000 7f664d71b700 1 -- 172.21.15.104:0/3082021279 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664809f040 con 0x7f6648132680 2024-09-18T14:21:38.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664d71b700 1 -- 172.21.15.104:0/3082021279 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648132680 msgr2=0x7f6648138ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664d71b700 1 --2- 172.21.15.104:0/3082021279 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648132680 0x7f6648138ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6640009e60 tx=0x7f6640009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2e33e815ee18f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 1 -- 172.21.15.104:0/3082021279 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664000e070 con 0x7f6648132680 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: dump: 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.641 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 1 -- 172.21.15.104:0/3082021279 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664000bd70 con 0x7f6648132680 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 5 monclient: authenticate success, global_id 4310 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 1 -- 172.21.15.104:0/3082021279 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640012660 con 0x7f6648132680 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 4 set_mon_vals no callback set 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f66477fe700 10 monclient: dump: 2024-09-18T14:21:38.642 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 10 monclient: shutdown 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 -- 172.21.15.104:0/3082021279 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648132680 msgr2=0x7f6648138ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 --2- 172.21.15.104:0/3082021279 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648132680 0x7f6648138ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6640009e60 tx=0x7f6640009780 comp rx=0 tx=0).stop 2024-09-18T14:21:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 -- 172.21.15.104:0/3082021279 shutdown_connections 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 --2- 172.21.15.104:0/3082021279 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648132680 0x7f6648138ed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 --2- 172.21.15.104:0/3082021279 >> v2:172.21.15.174:3300/0 conn(0x7f664806f440 0x7f664806b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 -- 172.21.15.104:0/3082021279 >> 172.21.15.104:0/3082021279 conn(0x7f66480682c0 msgr2=0x7f66480a30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 -- 172.21.15.104:0/3082021279 shutdown_connections 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.628+0000 7f664df1c700 1 -- 172.21.15.104:0/3082021279 wait complete. 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient: monmap: 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:38.645 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:38.645 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:38.645 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 Processor -- start 2024-09-18T14:21:38.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 -- start start 2024-09-18T14:21:38.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient: init 2024-09-18T14:21:38.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648131b70 0x7f6648134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): picked mon.noname-c con 0x7f6648131b70 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6648132680 0x7f6648135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): picked mon.noname-a con 0x7f6648132680 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648138ec0 0x7f6648135670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): picked mon.noname-b con 0x7f6648138ec0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648069e30 con 0x7f6648132680 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648131600 con 0x7f6648138ec0 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6648138570 con 0x7f6648131b70 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664df1c700 10 monclient(hunting): authenticate will time out at 819.627258s 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648131b70 0x7f6648134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6648132680 0x7f6648135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648131b70 0x7f6648134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35960/0 (socket says 172.21.15.104:35960) 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 1 -- 172.21.15.104:0/1091613414 learned_addr learned my addr 172.21.15.104:0/1091613414 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 10 monclient(hunting): get_auth_request con 0x7f6648131b70 auth_method 0 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): get_auth_request con 0x7f6648132680 auth_method 0 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664cf1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648138ec0 0x7f6648135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 10 monclient(hunting): get_auth_request con 0x7f6648138ec0 auth_method 0 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f664d71b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 1 -- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648131b70 msgr2=0x7f6648134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.632+0000 7f6647fff700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648131b70 0x7f6648134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 1 -- 172.21.15.104:0/1091613414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648138ec0 msgr2=0x7f6648135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648138ec0 0x7f6648135670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 1 monclient: found mon.noname-a 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 1 -- 172.21.15.104:0/1091613414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640009a20 con 0x7f6648132680 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664cf1a700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648131b70 0x7f6648134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 1 -- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6648132680 msgr2=0x7f6648135130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6647fff700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6648132680 0x7f6648135130 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f663800c560 tx=0x7f6638007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 1 -- 172.21.15.104:0/1091613414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638005d40 con 0x7f6648132680 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: dump: 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.654 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 1 -- 172.21.15.104:0/1091613414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66380070d0 con 0x7f6648132680 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664df1c700 5 monclient: authenticate success, global_id 4317 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664df1c700 10 monclient: _renew_subs 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664df1c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66481366d0 con 0x7f6648132680 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 1 -- 172.21.15.104:0/1091613414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638007490 con 0x7f6648132680 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6644ff9700 10 monclient: dump: 2024-09-18T14:21:38.655 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f6646ffd700 4 set_mon_vals no callback set 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664df1c700 10 monclient: _renew_subs 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664df1c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.636+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66481372a0 con 0x7f6648132680 2024-09-18T14:21:38.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f664df1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f664df1c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f664df1c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6648138700 con 0x7f6648132680 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f6644ff9700 1 -- 172.21.15.104:0/1091613414 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f663801c440 con 0x7f6648132680 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f6644ff9700 1 --2- 172.21.15.104:0/1091613414 >> v2:172.21.15.174:6812/17603 conn(0x7f6634035ab0 0x7f6634037f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f6644ff9700 1 -- 172.21.15.104:0/1091613414 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f66380472a0 con 0x7f6648132680 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f664cf1a700 1 --2- 172.21.15.104:0/1091613414 >> v2:172.21.15.174:6812/17603 conn(0x7f6634035ab0 0x7f6634037f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.640+0000 7f664cf1a700 10 monclient: get_auth_request con 0x7f6634035ab0 auth_method 0 2024-09-18T14:21:38.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.644+0000 7f664cf1a700 1 --2- 172.21.15.104:0/1091613414 >> v2:172.21.15.174:6812/17603 conn(0x7f6634035ab0 0x7f6634037f60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f663c009fd0 tx=0x7f663c0091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.648+0000 7f6644ff9700 1 -- 172.21.15.104:0/1091613414 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f663800fcd0 con 0x7f6648132680 2024-09-18T14:21:38.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.648+0000 7f6644ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.648+0000 7f6644ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:38.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.808+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66480633b0 con 0x7f6634035ab0 2024-09-18T14:21:38.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f6644ff9700 1 -- 172.21.15.104:0/1091613414 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7f66480633b0 con 0x7f6634035ab0 2024-09-18T14:21:38.820 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:38.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 >> v2:172.21.15.174:6812/17603 conn(0x7f6634035ab0 msgr2=0x7f6634037f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 --2- 172.21.15.104:0/1091613414 >> v2:172.21.15.174:6812/17603 conn(0x7f6634035ab0 0x7f6634037f60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f663c009fd0 tx=0x7f663c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 10 monclient: shutdown 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6648132680 msgr2=0x7f6648135130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6648132680 0x7f6648135130 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f663800c560 tx=0x7f6638007f90 comp rx=0 tx=0).stop 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 shutdown_connections 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 --2- 172.21.15.104:0/1091613414 >> v2:172.21.15.174:6812/17603 conn(0x7f6634035ab0 0x7f6634037f60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648138ec0 0x7f6648135670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6648132680 0x7f6648135130 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 --2- 172.21.15.104:0/1091613414 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6648131b70 0x7f6648134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.812+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 >> 172.21.15.104:0/1091613414 conn(0x7f66480682c0 msgr2=0x7f664806d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.816+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 shutdown_connections 2024-09-18T14:21:38.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.816+0000 7f664df1c700 1 -- 172.21.15.104:0/1091613414 wait complete. 2024-09-18T14:21:38.824 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:21:38.834 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-18T14:21:38.522903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4778452,"kb_used_data":557418280,"kb_used_omap":444,"kb_used_meta":0,"kb_avail":557417828,"statfs":{"total":575688990720,"available":570795855872,"internally_reserved":462848,"allocated":570796318720,"data_stored":570796318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":455649,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.119266"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135537+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:21:35.135537+0000","last_peered":"2024-09-18T14:21:35.135537+0000","last_clean":"2024-09-18T14:21:35.135537+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:21:35.135537+0000","last_undegraded":"2024-09-18T14:21:35.135537+0000","last_fullsized":"2024-09-18T14:21:35.135537+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295699+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:21:34.295699+0000","last_peered":"2024-09-18T14:21:34.295699+0000","last_clean":"2024-09-18T14:21:34.295699+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:21:34.295699+0000","last_undegraded":"2024-09-18T14:21:34.295699+0000","last_fullsized":"2024-09-18T14:21:34.295699+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135401+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:21:35.135401+0000","last_peered":"2024-09-18T14:21:35.135401+0000","last_clean":"2024-09-18T14:21:35.135401+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:21:35.135401+0000","last_undegraded":"2024-09-18T14:21:35.135401+0000","last_fullsized":"2024-09-18T14:21:35.135401+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295840+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:21:34.295840+0000","last_peered":"2024-09-18T14:21:34.295840+0000","last_clean":"2024-09-18T14:21:34.295840+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:21:34.295840+0000","last_undegraded":"2024-09-18T14:21:34.295840+0000","last_fullsized":"2024-09-18T14:21:34.295840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296315+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:21:34.296315+0000","last_peered":"2024-09-18T14:21:34.296315+0000","last_clean":"2024-09-18T14:21:34.296315+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:21:34.296315+0000","last_undegraded":"2024-09-18T14:21:34.296315+0000","last_fullsized":"2024-09-18T14:21:34.296315+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.349394+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:21:34.349394+0000","last_peered":"2024-09-18T14:21:34.349394+0000","last_clean":"2024-09-18T14:21:34.349394+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:21:34.349394+0000","last_undegraded":"2024-09-18T14:21:34.349394+0000","last_fullsized":"2024-09-18T14:21:34.349394+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135845+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:21:35.135845+0000","last_peered":"2024-09-18T14:21:35.135845+0000","last_clean":"2024-09-18T14:21:35.135845+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:21:35.135845+0000","last_undegraded":"2024-09-18T14:21:35.135845+0000","last_fullsized":"2024-09-18T14:21:35.135845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296035+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:21:34.296035+0000","last_peered":"2024-09-18T14:21:34.296035+0000","last_clean":"2024-09-18T14:21:34.296035+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:21:34.296035+0000","last_undegraded":"2024-09-18T14:21:34.296035+0000","last_fullsized":"2024-09-18T14:21:34.296035+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T14:21:33.278313+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:21:33.278313+0000","last_peered":"2024-09-18T14:21:33.278313+0000","last_clean":"2024-09-18T14:21:33.278313+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:21:33.278313+0000","last_undegraded":"2024-09-18T14:21:33.278313+0000","last_fullsized":"2024-09-18T14:21:33.278313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796012,"kb_used_data":92903436,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903368,"statfs":{"total":95948165120,"available":95133048832,"internally_reserved":69632,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58529,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795964,"kb_used_data":92903460,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903416,"statfs":{"total":95948165120,"available":95133097984,"internally_reserved":45056,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48713,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797228,"kb_used_data":92902244,"kb_used_omap":110,"kb_used_meta":0,"kb_avail":92902152,"statfs":{"total":95948165120,"available":95131803648,"internally_reserved":94208,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":113606,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795980,"kb_used_data":92903460,"kb_used_omap":54,"kb_used_meta":0,"kb_avail":92903400,"statfs":{"total":95948165120,"available":95133081600,"internally_reserved":61440,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55928,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796068,"kb_used_data":92903436,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":92903312,"statfs":{"total":95948165120,"available":95132991488,"internally_reserved":126976,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77686,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797200,"kb_used_data":92902244,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92902180,"statfs":{"total":95948165120,"available":95131832320,"internally_reserved":65536,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101187,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]}]}],"pool_statfs":[]}} 2024-09-18T14:21:38.835 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T14:21:38.836 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T14:21:38.836 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T14:21:38.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T14:21:38.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.924+0000 7f9c948ae700 10 monclient: get_monmap_and_config 2024-09-18T14:21:38.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.924+0000 7f9c948ae700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.924+0000 7f9c948ae700 10 monclient: monmap: 2024-09-18T14:21:38.934 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.934 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.934 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:38.931400+0000 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:38.931400+0000 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:38.935 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 1 Processor -- start 2024-09-18T14:21:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 1 -- start start 2024-09-18T14:21:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient: init 2024-09-18T14:21:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T14:21:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9c9009fe10 0x7f9c900a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient(hunting): picked mon.noname-c con 0x7f9c9009fe10 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c9009c900 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009c090 0x7f9c900a0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient(hunting): picked mon.noname-b con 0x7f9c9009c090 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9009b990 con 0x7f9c9009c900 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c900a1320 con 0x7f9c9009fe10 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900a0c60 con 0x7f9c9009c090 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c948ae700 10 monclient(hunting): authenticate will time out at 819.921263s 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009c090 0x7f9c900a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9c9009fe10 0x7f9c900a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8effd700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f9c9009c900 0x7f9c90131be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.104:36518/0 (socket says 172.21.15.104:36518) 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9c9009fe10 0x7f9c900a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.104:38896/0 (socket says 172.21.15.104:38896) 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8effd700 1 -- 172.21.15.104:0/1832637998 learned_addr learned my addr 172.21.15.104:0/1832637998 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): get_auth_request con 0x7f9c9009fe10 auth_method 0 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009c090 0x7f9c900a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35964/0 (socket says 172.21.15.104:35964) 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 10 monclient(hunting): get_auth_request con 0x7f9c9009c090 auth_method 0 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient(hunting): handle_auth_done global_id 4327 payload 274 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 1 -- 172.21.15.104:0/1832637998 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009c090 msgr2=0x7f9c900a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 1 --2- 172.21.15.104:0/1832637998 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009c090 0x7f9c900a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 1 -- 172.21.15.104:0/1832637998 >> v1:172.21.15.174:6789/0 conn(0x7f9c9009c900 legacy=0x7f9c90131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 1 monclient: found mon.noname-c 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8f7fe700 1 -- 172.21.15.104:0/1832637998 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9009b6b0 con 0x7f9c9009fe10 2024-09-18T14:21:38.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.928+0000 7f9c8ffff700 1 --2- 172.21.15.104:0/1832637998 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009c090 0x7f9c900a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:38.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8f7fe700 1 -- 172.21.15.104:0/1832637998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009fe10 msgr2=0x7f9c900a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8f7fe700 1 --2- 172.21.15.104:0/1832637998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009fe10 0x7f9c900a01e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098e0 tx=0x7f9c840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95b6fc0669de927e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/1832637998 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8400b850 con 0x7f9c9009fe10 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: dump: 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.945 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 5 monclient: authenticate success, global_id 4327 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/1832637998 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8400b9b0 con 0x7f9c9009fe10 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 4 set_mon_vals no callback set 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/1832637998 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8400bd50 con 0x7f9c9009fe10 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8e7fc700 10 monclient: dump: 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient: shutdown 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:38.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- 172.21.15.104:0/1832637998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009fe10 msgr2=0x7f9c900a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 --2- 172.21.15.104:0/1832637998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009fe10 0x7f9c900a01e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098e0 tx=0x7f9c840091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- 172.21.15.104:0/1832637998 shutdown_connections 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 --2- 172.21.15.104:0/1832637998 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009c090 0x7f9c900a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 --2- 172.21.15.104:0/1832637998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009fe10 0x7f9c900a01e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- 172.21.15.104:0/1832637998 >> 172.21.15.104:0/1832637998 conn(0x7f9c900a6080 msgr2=0x7f9c900676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- 172.21.15.104:0/1832637998 shutdown_connections 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- 172.21.15.104:0/1832637998 wait complete. 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient: build_initial_monmap 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient: monmap: 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:38.948 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 Processor -- start 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- start start 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient: init 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009c090 0x7f9c901414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): picked mon.noname-b con 0x7f9c9009c090 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:38.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 0x7f9c90141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): picked mon.noname-a con 0x7f9c9009c900 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009fe10 0x7f9c90141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): picked mon.noname-c con 0x7f9c9009fe10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90121d80 con 0x7f9c9009c900 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c901222b0 con 0x7f9c9009c090 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9009ae40 con 0x7f9c9009fe10 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c948ae700 10 monclient(hunting): authenticate will time out at 819.927705s 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 0x7f9c90141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 0x7f9c90141a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51144/0 (socket says 172.21.15.104:51144) 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8effd700 1 -- 172.21.15.104:0/3525433375 learned_addr learned my addr 172.21.15.104:0/3525433375 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8effd700 10 monclient(hunting): get_auth_request con 0x7f9c9009c900 auth_method 0 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.932+0000 7f9c8ffff700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009fe10 0x7f9c90141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8f7fe700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009c090 0x7f9c901414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8ffff700 10 monclient(hunting): get_auth_request con 0x7f9c9009fe10 auth_method 0 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8f7fe700 10 monclient(hunting): get_auth_request con 0x7f9c9009c090 auth_method 0 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 -- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009fe10 msgr2=0x7f9c90141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009fe10 0x7f9c90141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 -- 172.21.15.104:0/3525433375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009c090 msgr2=0x7f9c901414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:38.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009c090 0x7f9c901414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:38.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 10 monclient: _finish_hunting 0 2024-09-18T14:21:38.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 monclient: found mon.noname-a 2024-09-18T14:21:38.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:38.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 -- 172.21.15.104:0/3525433375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c80009420 con 0x7f9c9009c900 2024-09-18T14:21:38.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8f7fe700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009c090 0x7f9c901414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 -- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 msgr2=0x7f9c90141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8effd700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 0x7f9c90141a20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800ccd0 tx=0x7f9c88005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 1 -- 172.21.15.104:0/3525433375 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8800ad40 con 0x7f9c9009c900 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: dump: 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.955 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c948ae700 5 monclient: authenticate success, global_id 4320 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 1 -- 172.21.15.104:0/3525433375 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c880070f0 con 0x7f9c9009c900 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 1 -- 172.21.15.104:0/3525433375 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c880074b0 con 0x7f9c9009c900 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c777fe700 10 monclient: dump: 2024-09-18T14:21:38.956 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c8dffb700 4 set_mon_vals no callback set 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c948ae700 10 monclient: _renew_subs 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c948ae700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c84009420 con 0x7f9c9009c900 2024-09-18T14:21:38.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c948ae700 10 monclient: _renew_subs 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c948ae700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.936+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c90120a60 con 0x7f9c9009c900 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c777fe700 1 -- 172.21.15.104:0/3525433375 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f9c880078a0 con 0x7f9c9009c900 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c948ae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c948ae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c777fe700 1 --2- 172.21.15.104:0/3525433375 >> v2:172.21.15.174:6812/17603 conn(0x7f9c7c035a80 0x7f9c7c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c948ae700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c777fe700 1 -- 172.21.15.104:0/3525433375 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f9c880484e0 con 0x7f9c9009c900 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c900633b0 con 0x7f9c9009c900 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c8f7fe700 1 --2- 172.21.15.104:0/3525433375 >> v2:172.21.15.174:6812/17603 conn(0x7f9c7c035a80 0x7f9c7c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.940+0000 7f9c8f7fe700 10 monclient: get_auth_request con 0x7f9c7c035a80 auth_method 0 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.944+0000 7f9c8f7fe700 1 --2- 172.21.15.104:0/3525433375 >> v2:172.21.15.174:6812/17603 conn(0x7f9c7c035a80 0x7f9c7c037f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9c840030c0 tx=0x7f9c84016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:38.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.948+0000 7f9c777fe700 1 -- 172.21.15.104:0/3525433375 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9c8801b400 con 0x7f9c9009c900 2024-09-18T14:21:38.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.948+0000 7f9c777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:38.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:38.948+0000 7f9c777fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:39.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.140+0000 7f9c948ae700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-18T14:21:39.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.140+0000 7f9c948ae700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-18T14:21:39.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.140+0000 7f9c948ae700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:39.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.140+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9c90120150 con 0x7f9c9009c900 2024-09-18T14:21:39.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.140+0000 7f9c777fe700 1 -- 172.21.15.104:0/3525433375 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f9c88029040 con 0x7f9c9009c900 2024-09-18T14:21:39.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.140+0000 7f9c777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-18T14:21:39.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.140+0000 7f9c777fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:39.151 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:39.151 INFO:teuthology.orchestra.run.smithi104.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 >> v2:172.21.15.174:6812/17603 conn(0x7f9c7c035a80 msgr2=0x7f9c7c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 --2- 172.21.15.104:0/3525433375 >> v2:172.21.15.174:6812/17603 conn(0x7f9c7c035a80 0x7f9c7c037f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9c840030c0 tx=0x7f9c84016000 comp rx=0 tx=0).stop 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 10 monclient: shutdown 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 msgr2=0x7f9c90141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 0x7f9c90141a20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800ccd0 tx=0x7f9c88005480 comp rx=0 tx=0).stop 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 shutdown_connections 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 --2- 172.21.15.104:0/3525433375 >> v2:172.21.15.174:6812/17603 conn(0x7f9c7c035a80 0x7f9c7c037f30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c9009fe10 0x7f9c90141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9009c900 0x7f9c90141a20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 --2- 172.21.15.104:0/3525433375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c9009c090 0x7f9c901414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 >> 172.21.15.104:0/3525433375 conn(0x7f9c900a6080 msgr2=0x7f9c9009cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:39.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 shutdown_connections 2024-09-18T14:21:39.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.144+0000 7f9c948ae700 1 -- 172.21.15.104:0/3525433375 wait complete. 2024-09-18T14:21:39.169 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T14:21:39.169 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-18T14:21:39.183 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-18T14:21:39.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:21:39.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.280+0000 7f3cb3588700 10 monclient: get_monmap_and_config 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.280+0000 7f3cb3588700 10 monclient: build_initial_monmap 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.280+0000 7f3cb3588700 10 monclient: monmap: 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:39.287041+0000 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:39.287041+0000 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:39.290 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:39.291 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:39.291 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.280+0000 7f3cb3588700 1 Processor -- start 2024-09-18T14:21:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 1 -- start start 2024-09-18T14:21:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient: init 2024-09-18T14:21:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T14:21:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cac09a5b0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3cac09ae20 0x7f3cac130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient(hunting): picked mon.noname-a con 0x7f3cac09ae20 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:39.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cac134ec0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:39.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac09b220 con 0x7f3cac09a5b0 2024-09-18T14:21:39.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac098470 con 0x7f3cac134ec0 2024-09-18T14:21:39.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac099650 con 0x7f3cac09ae20 2024-09-18T14:21:39.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb3588700 10 monclient(hunting): authenticate will time out at 820.276597s 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3cac09ae20 0x7f3cac130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb2586700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3cac09a5b0 0x7f3cac12f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47518/0 (socket says 172.21.15.104:47518) 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3cac09ae20 0x7f3cac130090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51158/0 (socket says 172.21.15.104:51158) 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb2586700 1 -- 172.21.15.104:0/1002320818 learned_addr learned my addr 172.21.15.104:0/1002320818 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): get_auth_request con 0x7f3cac09ae20 auth_method 0 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2024-09-18T14:21:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 -- 172.21.15.104:0/1002320818 >> v1:172.21.15.174:6789/0 conn(0x7f3cac134ec0 legacy=0x7f3cac1372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 -- 172.21.15.104:0/1002320818 >> v1:172.21.15.104:6789/0 conn(0x7f3cac09a5b0 legacy=0x7f3cac12f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient: _finish_hunting 0 2024-09-18T14:21:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 monclient: found mon.noname-a 2024-09-18T14:21:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 -- 172.21.15.104:0/1002320818 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac099930 con 0x7f3cac09ae20 2024-09-18T14:21:39.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 -- 172.21.15.104:0/1002320818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac09ae20 msgr2=0x7f3cac130090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:39.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.284+0000 7f3cb1d85700 1 --2- 172.21.15.104:0/1002320818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac09ae20 0x7f3cac130090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80098e0 tx=0x7f3ca80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c5aa51e94b95ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:39.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 1 -- 172.21.15.104:0/1002320818 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800ba70 con 0x7f3cac09ae20 2024-09-18T14:21:39.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: dump: 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: _finish_auth 0 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:39.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 1 -- 172.21.15.104:0/1002320818 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca800bbd0 con 0x7f3cac09ae20 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 5 monclient: authenticate success, global_id 4323 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 1 -- 172.21.15.104:0/1002320818 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8011420 con 0x7f3cac09ae20 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 4 set_mon_vals no callback set 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb1584700 10 monclient: dump: 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.300 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 10 monclient: shutdown 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 -- 172.21.15.104:0/1002320818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac09ae20 msgr2=0x7f3cac130090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 --2- 172.21.15.104:0/1002320818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac09ae20 0x7f3cac130090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80098e0 tx=0x7f3ca80091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 -- 172.21.15.104:0/1002320818 shutdown_connections 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 --2- 172.21.15.104:0/1002320818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac09ae20 0x7f3cac130090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 -- 172.21.15.104:0/1002320818 >> 172.21.15.104:0/1002320818 conn(0x7f3cac0a12c0 msgr2=0x7f3cac0a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 -- 172.21.15.104:0/1002320818 shutdown_connections 2024-09-18T14:21:39.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 -- 172.21.15.104:0/1002320818 wait complete. 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 10 monclient: build_initial_monmap 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 10 monclient: monmap: 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 Processor -- start 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 1 -- start start 2024-09-18T14:21:39.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.288+0000 7f3cb3588700 10 monclient: init 2024-09-18T14:21:39.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:39.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:39.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3cac12c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): picked mon.noname-b con 0x7f3cac09a5b0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:39.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3cac06db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): picked mon.noname-c con 0x7f3cac09ae20 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): picked mon.noname-a con 0x7f3cac134ec0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac135770 con 0x7f3cac134ec0 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac118630 con 0x7f3cac09a5b0 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac1179c0 con 0x7f3cac09ae20 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb3588700 10 monclient(hunting): authenticate will time out at 820.285055s 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51168/0 (socket says 172.21.15.104:51168) 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 -- 172.21.15.104:0/153815758 learned_addr learned my addr 172.21.15.104:0/153815758 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): get_auth_request con 0x7f3cac134ec0 auth_method 0 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3cac06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 10 monclient(hunting): get_auth_request con 0x7f3cac09ae20 auth_method 0 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2586700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3cac12c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2586700 10 monclient(hunting): get_auth_request con 0x7f3cac09a5b0 auth_method 0 2024-09-18T14:21:39.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2586700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2586700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2586700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb1d85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 msgr2=0x7f3cac06db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3cac06db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 msgr2=0x7f3cac12c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3cac12c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient: _finish_hunting 0 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 monclient: found mon.noname-a 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8009420 con 0x7f3cac134ec0 2024-09-18T14:21:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2586700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3cac12c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:21:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 msgr2=0x7f3cac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:21:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 msgr2=0x7f3cac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:21:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400c890 tx=0x7f3ca40026a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.292+0000 7f3cb2d87700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient: get_auth_request con 0x7f3cac134ec0 auth_method 0 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 0 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: ms_handle_reset current mon [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 msgr2=0x7f3cac06e0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cac134ec0 0x7f3cac06e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3c90005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient(hunting): picked mon.noname-c con 0x7f3cac09ae20 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 0x7f3c90008370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient(hunting): picked mon.noname-a con 0x7f3c90005ef0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3c9000aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient(hunting): picked mon.noname-b con 0x7f3cac09a5b0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9000b870 con 0x7f3c90005ef0 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9000b9b0 con 0x7f3cac09a5b0 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9000bb70 con 0x7f3cac09ae20 2024-09-18T14:21:39.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3c90005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 0x7f3c90008370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): get_auth_request con 0x7f3cac09ae20 auth_method 0 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 10 monclient(hunting): get_auth_request con 0x7f3c90005ef0 auth_method 0 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 msgr2=0x7f3c90008370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 msgr2=0x7f3c90008370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 0x7f3c90008370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3c9000aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb1d85700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 0x7f3c90008370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient(hunting): get_auth_request con 0x7f3cac09a5b0 auth_method 0 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2d87700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient(hunting): handle_auth_done global_id 4326 payload 293 2024-09-18T14:21:39.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 msgr2=0x7f3c9000aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3c9000aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 msgr2=0x7f3c90008370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 0x7f3c90008370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient: _finish_hunting 0 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 monclient: found mon.noname-c 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9000bdf0 con 0x7f3cac09ae20 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 msgr2=0x7f3c90005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb2586700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3c90005520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c0053c0 tx=0x7f3c9c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c004d00 con 0x7f3cac09ae20 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: dump: 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9c005b30 con 0x7f3cac09ae20 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb3588700 5 monclient: authenticate success, global_id 4326 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb3588700 10 monclient: _renew_subs 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb3588700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac118980 con 0x7f3cac09ae20 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c0070d0 con 0x7f3cac09ae20 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3ca27fc700 10 monclient: dump: 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb0d83700 4 set_mon_vals no callback set 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb3588700 10 monclient: _renew_subs 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb3588700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.496+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac06f6f0 con 0x7f3cac09ae20 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f3c9c007480 con 0x7f3cac09ae20 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3cb3588700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3cb3588700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3cb3588700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac0633b0 con 0x7f3cac09ae20 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3ca27fc700 1 --2- 172.21.15.104:0/153815758 >> v2:172.21.15.174:6812/17603 conn(0x7f3cac134ec0 0x7f3c90040170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f3c9c046880 con 0x7f3cac09ae20 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3cb1d85700 1 --2- 172.21.15.104:0/153815758 >> v2:172.21.15.174:6812/17603 conn(0x7f3cac134ec0 0x7f3c90040170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.500+0000 7f3cb1d85700 10 monclient: get_auth_request con 0x7f3cac134ec0 auth_method 0 2024-09-18T14:21:39.513 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.504+0000 7f3cb1d85700 1 --2- 172.21.15.104:0/153815758 >> v2:172.21.15.174:6812/17603 conn(0x7f3cac134ec0 0x7f3c90040170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80093f0 tx=0x7f3ca8016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.508+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3c9c00e550 con 0x7f3cac09ae20 2024-09-18T14:21:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.508+0000 7f3ca27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.508+0000 7f3ca27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:39.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.680+0000 7f3cb3588700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:39.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.680+0000 7f3cb3588700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:39.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.680+0000 7f3cb3588700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.680+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cac1190e0 con 0x7f3cac09ae20 2024-09-18T14:21:39.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.680+0000 7f3ca27fc700 1 -- 172.21.15.104:0/153815758 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+10065 (secure 0 0 0) 0x7f3c9c012380 con 0x7f3cac09ae20 2024-09-18T14:21:39.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.680+0000 7f3ca27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:21:39.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.680+0000 7f3ca27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:39.690 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":20,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:21:38.511917+0000","last_up_change":"2024-09-18T14:21:25.123789+0000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T14:21:26.474399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T14:21:30.433492+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":17763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":17763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":17763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":17763}]},"public_addr":"172.21.15.104:6808/17763","cluster_addr":"172.21.15.104:6809/17763","heartbeat_back_addr":"172.21.15.104:6811/17763","heartbeat_front_addr":"172.21.15.104:6810/17763","state":["exists","up"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":17761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":17761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":17761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":17761}]},"public_addr":"172.21.15.104:6804/17761","cluster_addr":"172.21.15.104:6805/17761","heartbeat_back_addr":"172.21.15.104:6807/17761","heartbeat_front_addr":"172.21.15.104:6806/17761","state":["exists","up"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":17762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":17762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":17762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":17762}]},"public_addr":"172.21.15.104:6800/17762","cluster_addr":"172.21.15.104:6801/17762","heartbeat_back_addr":"172.21.15.104:6803/17762","heartbeat_front_addr":"172.21.15.104:6802/17762","state":["exists","up"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":17845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":17845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":17845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":17845}]},"public_addr":"172.21.15.174:6808/17845","cluster_addr":"172.21.15.174:6809/17845","heartbeat_back_addr":"172.21.15.174:6811/17845","heartbeat_front_addr":"172.21.15.174:6810/17845","state":["exists","up"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":17847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6801","nonce":17847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":17847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":17847}]},"public_addr":"172.21.15.174:6800/17847","cluster_addr":"172.21.15.174:6801/17847","heartbeat_back_addr":"172.21.15.174:6803/17847","heartbeat_front_addr":"172.21.15.174:6802/17847","state":["exists","up"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":17846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":17846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":17846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":17846}]},"public_addr":"172.21.15.174:6804/17846","cluster_addr":"172.21.15.174:6805/17846","heartbeat_back_addr":"172.21.15.174:6807/17846","heartbeat_front_addr":"172.21.15.174:6806/17846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.237413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.242543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.214389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.213881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.183415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.230911+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:21:39.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 >> v2:172.21.15.174:6812/17603 conn(0x7f3cac134ec0 msgr2=0x7f3c90040170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 --2- 172.21.15.104:0/153815758 >> v2:172.21.15.174:6812/17603 conn(0x7f3cac134ec0 0x7f3c90040170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80093f0 tx=0x7f3ca8016000 comp rx=0 tx=0).stop 2024-09-18T14:21:39.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 10 monclient: shutdown 2024-09-18T14:21:39.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 msgr2=0x7f3c90005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3c90005520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c0053c0 tx=0x7f3c9c009040 comp rx=0 tx=0).stop 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 shutdown_connections 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 --2- 172.21.15.104:0/153815758 >> v2:172.21.15.174:6812/17603 conn(0x7f3cac134ec0 0x7f3c90040170 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cac09a5b0 0x7f3c9000aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c90005ef0 0x7f3c90008370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 --2- 172.21.15.104:0/153815758 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3cac09ae20 0x7f3c90005520 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 >> 172.21.15.104:0/153815758 conn(0x7f3cac0a12c0 msgr2=0x7f3cac12e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 shutdown_connections 2024-09-18T14:21:39.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.684+0000 7f3cb3588700 1 -- 172.21.15.104:0/153815758 wait complete. 2024-09-18T14:21:39.703 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T14:21:26.474399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T14:21:30.433492+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-18T14:21:39.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.812+0000 7f3752c47700 10 monclient: get_monmap_and_config 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.812+0000 7f3752c47700 10 monclient: build_initial_monmap 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.812+0000 7f3752c47700 10 monclient: monmap: 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:39.819224+0000 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:39.819224+0000 2024-09-18T14:21:39.822 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:39.823 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.823 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:39.823 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:39.823 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:39.823 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:39.823 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:39.823 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 1 Processor -- start 2024-09-18T14:21:39.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 1 -- start start 2024-09-18T14:21:39.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient: init 2024-09-18T14:21:39.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:39.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f374c09a6b0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f374c09aa80 0x7f374c1322f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient(hunting): picked mon.noname-c con 0x7f374c09aa80 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 0x7f374c1392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient(hunting): picked mon.noname-b con 0x7f374c136f10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374c09b6e0 con 0x7f374c09a6b0 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374c0a1280 con 0x7f374c09aa80 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374c09b9c0 con 0x7f374c136f10 2024-09-18T14:21:39.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752c47700 10 monclient(hunting): authenticate will time out at 820.810036s 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 0x7f374c1392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f374c09aa80 0x7f374c1322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 0x7f374c1392f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:35996/0 (socket says 172.21.15.104:35996) 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f374c09aa80 0x7f374c1322f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.104:38934/0 (socket says 172.21.15.104:38934) 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 -- 172.21.15.104:0/3775728885 learned_addr learned my addr 172.21.15.104:0/3775728885 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): get_auth_request con 0x7f374c136f10 auth_method 0 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 10 monclient(hunting): get_auth_request con 0x7f374c09aa80 auth_method 0 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 -- 172.21.15.104:0/3775728885 >> v2:172.21.15.174:3300/0 conn(0x7f374c09aa80 msgr2=0x7f374c1322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 --2- 172.21.15.104:0/3775728885 >> v2:172.21.15.174:3300/0 conn(0x7f374c09aa80 0x7f374c1322f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 -- 172.21.15.104:0/3775728885 >> v1:172.21.15.174:6789/0 conn(0x7f374c09a6b0 legacy=0x7f374c131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient: _finish_hunting 0 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 monclient: found mon.noname-b 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:39.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3752446700 1 -- 172.21.15.104:0/3775728885 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c099e60 con 0x7f374c136f10 2024-09-18T14:21:39.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.816+0000 7f3751444700 1 --2- 172.21.15.104:0/3775728885 >> v2:172.21.15.174:3300/0 conn(0x7f374c09aa80 0x7f374c1322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:39.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752446700 1 -- 172.21.15.104:0/3775728885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 msgr2=0x7f374c1392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752446700 1 --2- 172.21.15.104:0/3775728885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 0x7f374c1392f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f374c06aae0 tx=0x7f3744009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d11c779a0388dbe server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 1 -- 172.21.15.104:0/3775728885 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374400e070 con 0x7f374c136f10 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: dump: 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.838 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: _finish_auth 0 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 5 monclient: authenticate success, global_id 4325 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 1 -- 172.21.15.104:0/3775728885 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374400bc30 con 0x7f374c136f10 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 4 set_mon_vals no callback set 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 1 -- 172.21.15.104:0/3775728885 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37440092e0 con 0x7f374c136f10 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:39.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3750c43700 10 monclient: dump: 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 10 monclient: shutdown 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:39.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 -- 172.21.15.104:0/3775728885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 msgr2=0x7f374c1392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 --2- 172.21.15.104:0/3775728885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 0x7f374c1392f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f374c06aae0 tx=0x7f3744009840 comp rx=0 tx=0).stop 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 -- 172.21.15.104:0/3775728885 shutdown_connections 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 --2- 172.21.15.104:0/3775728885 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c136f10 0x7f374c1392f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 --2- 172.21.15.104:0/3775728885 >> v2:172.21.15.174:3300/0 conn(0x7f374c09aa80 0x7f374c1322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 -- 172.21.15.104:0/3775728885 >> 172.21.15.104:0/3775728885 conn(0x7f374c0a53b0 msgr2=0x7f374c0a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 -- 172.21.15.104:0/3775728885 shutdown_connections 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.820+0000 7f3752c47700 1 -- 172.21.15.104:0/3775728885 wait complete. 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient: build_initial_monmap 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient: monmap: 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:39.841 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 Processor -- start 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 -- start start 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient: init 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 0x7f374c135930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): picked mon.noname-c con 0x7f374c09a6b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f374c09aa80 0x7f374c135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): picked mon.noname-b con 0x7f374c09aa80 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f374c136f10 0x7f374c1363b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): picked mon.noname-a con 0x7f374c136f10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c09aee0 con 0x7f374c136f10 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c09dc20 con 0x7f374c09aa80 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374c06d030 con 0x7f374c09a6b0 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752c47700 10 monclient(hunting): authenticate will time out at 820.817725s 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 0x7f374c135930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f374c136f10 0x7f374c1363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 0x7f374c135930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:36008/0 (socket says 172.21.15.104:36008) 2024-09-18T14:21:39.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 -- 172.21.15.104:0/2920277149 learned_addr learned my addr 172.21.15.104:0/2920277149 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): get_auth_request con 0x7f374c09a6b0 auth_method 0 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 10 monclient(hunting): get_auth_request con 0x7f374c136f10 auth_method 0 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751444700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f374c09aa80 0x7f374c135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3752446700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751444700 10 monclient(hunting): get_auth_request con 0x7f374c09aa80 auth_method 0 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751444700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751444700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751444700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 -- 172.21.15.104:0/2920277149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f374c09aa80 msgr2=0x7f374c135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f374c09aa80 0x7f374c135e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 -- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f374c136f10 msgr2=0x7f374c1363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f374c136f10 0x7f374c1363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient: _finish_hunting 0 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 monclient: found mon.noname-c 2024-09-18T14:21:39.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.824+0000 7f3751c45700 1 -- 172.21.15.104:0/2920277149 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c009420 con 0x7f374c09a6b0 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3751c45700 1 -- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 msgr2=0x7f374c135930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3751c45700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 0x7f374c135930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f374800c600 tx=0x7f3748007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 1 -- 172.21.15.104:0/2920277149 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37480117a0 con 0x7f374c09a6b0 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: dump: 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 5 monclient: authenticate success, global_id 4328 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 1 -- 172.21.15.104:0/2920277149 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3748011900 con 0x7f374c09a6b0 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 1 -- 172.21.15.104:0/2920277149 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748011cc0 con 0x7f374c09a6b0 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 10 monclient: _renew_subs 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3743fff700 4 set_mon_vals no callback set 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3744009ae0 con 0x7f374c09a6b0 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 10 monclient: dump: 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 10 monclient: _renew_subs 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374c06ce00 con 0x7f374c09a6b0 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 1 -- 172.21.15.104:0/2920277149 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f3748012630 con 0x7f374c09a6b0 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374c06dfc0 con 0x7f374c09a6b0 2024-09-18T14:21:39.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 1 --2- 172.21.15.104:0/2920277149 >> v2:172.21.15.174:6812/17603 conn(0x7f3730035a80 0x7f3730037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:39.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.828+0000 7f3741ffb700 1 -- 172.21.15.104:0/2920277149 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f3748047c10 con 0x7f374c09a6b0 2024-09-18T14:21:39.850 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.832+0000 7f3751444700 1 --2- 172.21.15.104:0/2920277149 >> v2:172.21.15.174:6812/17603 conn(0x7f3730035a80 0x7f3730037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.832+0000 7f3751444700 10 monclient: get_auth_request con 0x7f3730035a80 auth_method 0 2024-09-18T14:21:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.832+0000 7f3751444700 1 --2- 172.21.15.104:0/2920277149 >> v2:172.21.15.174:6812/17603 conn(0x7f3730035a80 0x7f3730037f30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f373c0093f0 tx=0x7f373c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.840+0000 7f3741ffb700 1 -- 172.21.15.104:0/2920277149 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3748011e20 con 0x7f374c09a6b0 2024-09-18T14:21:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.840+0000 7f3741ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:39.840+0000 7f3741ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:40.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T14:21:40.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T14:21:40.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:40.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f374c063340 con 0x7f374c09a6b0 2024-09-18T14:21:40.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3741ffb700 1 -- 172.21.15.104:0/2920277149 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37480102e0 con 0x7f374c09a6b0 2024-09-18T14:21:40.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3741ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T14:21:40.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3741ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:40.012 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-09-18T14:21:40.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 >> v2:172.21.15.174:6812/17603 conn(0x7f3730035a80 msgr2=0x7f3730037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 --2- 172.21.15.104:0/2920277149 >> v2:172.21.15.174:6812/17603 conn(0x7f3730035a80 0x7f3730037f30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f373c0093f0 tx=0x7f373c008040 comp rx=0 tx=0).stop 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 10 monclient: shutdown 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 msgr2=0x7f374c135930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 0x7f374c135930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f374800c600 tx=0x7f3748007f60 comp rx=0 tx=0).stop 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 shutdown_connections 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 --2- 172.21.15.104:0/2920277149 >> v2:172.21.15.174:6812/17603 conn(0x7f3730035a80 0x7f3730037f30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f374c136f10 0x7f374c1363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f374c09aa80 0x7f374c135e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 --2- 172.21.15.104:0/2920277149 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f374c09a6b0 0x7f374c135930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 >> 172.21.15.104:0/2920277149 conn(0x7f374c0a53b0 msgr2=0x7f374c09e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:40.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 shutdown_connections 2024-09-18T14:21:40.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.004+0000 7f3752c47700 1 -- 172.21.15.104:0/2920277149 wait complete. 2024-09-18T14:21:40.026 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-18T14:21:40.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient: get_monmap_and_config 2024-09-18T14:21:40.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient: build_initial_monmap 2024-09-18T14:21:40.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient: monmap: 2024-09-18T14:21:40.140 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:40.140 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.140 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:40.137390+0000 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:40.137390+0000 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:40.141 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 1 Processor -- start 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 1 -- start start 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient: init 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f79b812f960 0x7f79b812fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f79b812f960 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79b809bc00 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f79b8130270 0x7f79b812ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f79b8130270 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b8099c10 con 0x7f79b809bc00 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b809ad90 con 0x7f79b812f960 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b8099930 con 0x7f79b8130270 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b7fff700 10 monclient(hunting): authenticate will time out at 821.127014s 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f79b812f960 0x7f79b812fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f79b812f960 0x7f79b812fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38296/0 (socket says 172.21.15.104:38296) 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 -- 172.21.15.104:0/2367643957 learned_addr learned my addr 172.21.15.104:0/2367643957 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): get_auth_request con 0x7f79b812f960 auth_method 0 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 1 --2- 172.21.15.104:0/2367643957 >> v2:172.21.15.174:3300/0 conn(0x7f79b8130270 0x7f79b812ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 10 monclient(hunting): get_auth_request con 0x7f79b8130270 auth_method 0 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 -- 172.21.15.104:0/2367643957 >> v2:172.21.15.174:3300/0 conn(0x7f79b8130270 msgr2=0x7f79b812ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 --2- 172.21.15.104:0/2367643957 >> v2:172.21.15.174:3300/0 conn(0x7f79b8130270 0x7f79b812ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 -- 172.21.15.104:0/2367643957 >> v1:172.21.15.174:6789/0 conn(0x7f79b809bc00 legacy=0x7f79b812c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient: _finish_hunting 0 2024-09-18T14:21:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 monclient: found mon.noname-a 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.132+0000 7f79b6ffd700 1 -- 172.21.15.104:0/2367643957 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b809d7c0 con 0x7f79b812f960 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b6ffd700 1 -- 172.21.15.104:0/2367643957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 msgr2=0x7f79b812fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b6ffd700 1 --2- 172.21.15.104:0/2367643957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812fd30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79b8098630 tx=0x7f79a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97318fd710f0ffc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 1 -- 172.21.15.104:0/2367643957 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8009e70 con 0x7f79b812f960 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: dump: 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.151 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 5 monclient: authenticate success, global_id 4332 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 1 -- 172.21.15.104:0/2367643957 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a800ba70 con 0x7f79b812f960 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 4 set_mon_vals no callback set 2024-09-18T14:21:40.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 1 -- 172.21.15.104:0/2367643957 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a800be30 con 0x7f79b812f960 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b67fc700 10 monclient: dump: 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.153 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 10 monclient: shutdown 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 -- 172.21.15.104:0/2367643957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 msgr2=0x7f79b812fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 --2- 172.21.15.104:0/2367643957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812fd30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79b8098630 tx=0x7f79a8009250 comp rx=0 tx=0).stop 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 -- 172.21.15.104:0/2367643957 shutdown_connections 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 --2- 172.21.15.104:0/2367643957 >> v2:172.21.15.174:3300/0 conn(0x7f79b8130270 0x7f79b812ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 --2- 172.21.15.104:0/2367643957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812fd30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 -- 172.21.15.104:0/2367643957 >> 172.21.15.104:0/2367643957 conn(0x7f79b80a11b0 msgr2=0x7f79b80a15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 -- 172.21.15.104:0/2367643957 shutdown_connections 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.136+0000 7f79b7fff700 1 -- 172.21.15.104:0/2367643957 wait complete. 2024-09-18T14:21:40.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient: build_initial_monmap 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient: monmap: 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 Processor -- start 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 -- start start 2024-09-18T14:21:40.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient: init 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79b809bc00 0x7f79b813cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f79b809bc00 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f79b812f960 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f79b8130270 0x7f79b8119720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f79b8130270 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b811b700 con 0x7f79b812f960 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b811b260 con 0x7f79b809bc00 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b809b460 con 0x7f79b8130270 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b7fff700 10 monclient(hunting): authenticate will time out at 821.134891s 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f79b8130270 0x7f79b8119720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f79b8130270 0x7f79b8119720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42324/0 (socket says 172.21.15.104:42324) 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38302/0 (socket says 172.21.15.104:38302) 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 1 -- 172.21.15.104:0/1418305259 learned_addr learned my addr 172.21.15.104:0/1418305259 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 10 monclient(hunting): get_auth_request con 0x7f79b8130270 auth_method 0 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): get_auth_request con 0x7f79b812f960 auth_method 0 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79b809bc00 0x7f79b813cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 10 monclient(hunting): get_auth_request con 0x7f79b809bc00 auth_method 0 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 -- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f79b8130270 msgr2=0x7f79b8119720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f79b8130270 0x7f79b8119720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 -- 172.21.15.104:0/1418305259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79b809bc00 msgr2=0x7f79b813cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79b809bc00 0x7f79b813cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient: _finish_hunting 0 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 monclient: found mon.noname-a 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:40.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.140+0000 7f79ae7fc700 1 -- 172.21.15.104:0/1418305259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a80094f0 con 0x7f79b812f960 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79ae7fc700 1 -- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 msgr2=0x7f79b812eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79ae7fc700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812eb80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c4e0 tx=0x7f79a0007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 1 -- 172.21.15.104:0/1418305259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a000ce90 con 0x7f79b812f960 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: dump: 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.164 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 5 monclient: authenticate success, global_id 4335 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 1 -- 172.21.15.104:0/1418305259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a00117a0 con 0x7f79b812f960 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 10 monclient: _renew_subs 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b811bdc0 con 0x7f79b812f960 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 1 -- 172.21.15.104:0/1418305259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0011b80 con 0x7f79b812f960 2024-09-18T14:21:40.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 10 monclient: dump: 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 10 monclient: _renew_subs 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b57fa700 4 set_mon_vals no callback set 2024-09-18T14:21:40.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b813dd80 con 0x7f79b812f960 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79af7fe700 1 -- 172.21.15.104:0/1418305259 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f79a001c460 con 0x7f79b812f960 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.144+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8063340 con 0x7f79b812f960 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.148+0000 7f79af7fe700 1 --2- 172.21.15.104:0/1418305259 >> v2:172.21.15.174:6812/17603 conn(0x7f7998035ad0 0x7f7998037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.148+0000 7f79af7fe700 1 -- 172.21.15.104:0/1418305259 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f79a0047ae0 con 0x7f79b812f960 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.148+0000 7f79b6ffd700 1 --2- 172.21.15.104:0/1418305259 >> v2:172.21.15.174:6812/17603 conn(0x7f7998035ad0 0x7f7998037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.148+0000 7f79b6ffd700 10 monclient: get_auth_request con 0x7f7998035ad0 auth_method 0 2024-09-18T14:21:40.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.148+0000 7f79b6ffd700 1 --2- 172.21.15.104:0/1418305259 >> v2:172.21.15.174:6812/17603 conn(0x7f7998035ad0 0x7f7998037f80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f79b8098630 tx=0x7f79a8016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.152+0000 7f79af7fe700 1 -- 172.21.15.104:0/1418305259 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f79a0049e60 con 0x7f79b812f960 2024-09-18T14:21:40.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.152+0000 7f79af7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.152+0000 7f79af7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:40.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T14:21:40.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T14:21:40.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f79b811d020 con 0x7f79b812f960 2024-09-18T14:21:40.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79af7fe700 1 -- 172.21.15.104:0/1418305259 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f79a0002090 con 0x7f79b812f960 2024-09-18T14:21:40.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79af7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T14:21:40.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79af7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:40.324 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2024-09-18T14:21:40.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 >> v2:172.21.15.174:6812/17603 conn(0x7f7998035ad0 msgr2=0x7f7998037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 1 --2- 172.21.15.104:0/1418305259 >> v2:172.21.15.174:6812/17603 conn(0x7f7998035ad0 0x7f7998037f80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f79b8098630 tx=0x7f79a8016000 comp rx=0 tx=0).stop 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 10 monclient: shutdown 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 msgr2=0x7f79b812eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.316+0000 7f79b7fff700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812eb80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c4e0 tx=0x7f79a0007ea0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 shutdown_connections 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 --2- 172.21.15.104:0/1418305259 >> v2:172.21.15.174:6812/17603 conn(0x7f7998035ad0 0x7f7998037f80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f79b8130270 0x7f79b8119720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79b812f960 0x7f79b812eb80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 --2- 172.21.15.104:0/1418305259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79b809bc00 0x7f79b813cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 >> 172.21.15.104:0/1418305259 conn(0x7f79b80a11b0 msgr2=0x7f79b8096180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 shutdown_connections 2024-09-18T14:21:40.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.320+0000 7f79b7fff700 1 -- 172.21.15.104:0/1418305259 wait complete. 2024-09-18T14:21:40.338 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-18T14:21:40.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:40.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.436+0000 7fdb03fff700 10 monclient: get_monmap_and_config 2024-09-18T14:21:40.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.436+0000 7fdb03fff700 10 monclient: build_initial_monmap 2024-09-18T14:21:40.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.436+0000 7fdb03fff700 10 monclient: monmap: 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:40.442515+0000 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:40.442515+0000 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:40.446 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:40.447 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:40.447 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:40.447 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:40.447 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.436+0000 7fdb03fff700 1 Processor -- start 2024-09-18T14:21:40.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.436+0000 7fdb03fff700 1 -- start start 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.436+0000 7fdb03fff700 10 monclient: init 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdb040a59d0 0x7fdb0409ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient(hunting): picked mon.noname-c con 0x7fdb040a59d0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb0409bea0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdb0409f240 0x7fdb040a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient(hunting): picked mon.noname-a con 0x7fdb0409f240 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:40.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0409af10 con 0x7fdb0409bea0 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb040a1870 con 0x7fdb0409f240 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb040a5da0 con 0x7fdb040a59d0 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb03fff700 10 monclient(hunting): authenticate will time out at 821.432638s 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdb0409f240 0x7fdb040a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdb0409f240 0x7fdb040a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38310/0 (socket says 172.21.15.104:38310) 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 -- 172.21.15.104:0/2756383652 learned_addr learned my addr 172.21.15.104:0/2756383652 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): get_auth_request con 0x7fdb0409f240 auth_method 0 2024-09-18T14:21:40.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 1 --2- 172.21.15.104:0/2756383652 >> v2:172.21.15.174:3300/0 conn(0x7fdb040a59d0 0x7fdb0409ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 10 monclient(hunting): get_auth_request con 0x7fdb040a59d0 auth_method 0 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb02ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 -- 172.21.15.104:0/2756383652 >> v2:172.21.15.174:3300/0 conn(0x7fdb040a59d0 msgr2=0x7fdb0409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 --2- 172.21.15.104:0/2756383652 >> v2:172.21.15.174:3300/0 conn(0x7fdb040a59d0 0x7fdb0409ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 -- 172.21.15.104:0/2756383652 >> v1:172.21.15.174:6789/0 conn(0x7fdb0409bea0 legacy=0x7fdb0409fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 monclient: found mon.noname-a 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.440+0000 7fdb037fe700 1 -- 172.21.15.104:0/2756383652 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0409cae0 con 0x7fdb0409f240 2024-09-18T14:21:40.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb037fe700 1 -- 172.21.15.104:0/2756383652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb0409f240 msgr2=0x7fdb040a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:40.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb037fe700 1 --2- 172.21.15.104:0/2756383652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb0409f240 0x7fdb040a03c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdb0409b980 tx=0x7fdafc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64a33b769ed45d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 1 -- 172.21.15.104:0/2756383652 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc00e070 con 0x7fdb0409f240 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: dump: 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 5 monclient: authenticate success, global_id 4338 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 1 -- 172.21.15.104:0/2756383652 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc00bc00 con 0x7fdb0409f240 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 4 set_mon_vals no callback set 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 1 -- 172.21.15.104:0/2756383652 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0092e0 con 0x7fdb0409f240 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb027fc700 10 monclient: dump: 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.466 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient: shutdown 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 -- 172.21.15.104:0/2756383652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb0409f240 msgr2=0x7fdb040a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 --2- 172.21.15.104:0/2756383652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb0409f240 0x7fdb040a03c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdb0409b980 tx=0x7fdafc009840 comp rx=0 tx=0).stop 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 -- 172.21.15.104:0/2756383652 shutdown_connections 2024-09-18T14:21:40.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 --2- 172.21.15.104:0/2756383652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb0409f240 0x7fdb040a03c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 --2- 172.21.15.104:0/2756383652 >> v2:172.21.15.174:3300/0 conn(0x7fdb040a59d0 0x7fdb0409ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 -- 172.21.15.104:0/2756383652 >> 172.21.15.104:0/2756383652 conn(0x7fdb04067c10 msgr2=0x7fdb040a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 -- 172.21.15.104:0/2756383652 shutdown_connections 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 -- 172.21.15.104:0/2756383652 wait complete. 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient: build_initial_monmap 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient: monmap: 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.468 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 Processor -- start 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 -- start start 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient: init 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb0409bea0 0x7fdb04140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 10 monclient(hunting): picked mon.noname-b con 0x7fdb0409bea0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.444+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 0x7fdb04131c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient(hunting): picked mon.noname-c con 0x7fdb0409f240 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb040a59d0 0x7fdb0411c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient(hunting): picked mon.noname-a con 0x7fdb040a59d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0406f3c0 con 0x7fdb040a59d0 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb04120340 con 0x7fdb0409bea0 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb04120060 con 0x7fdb0409f240 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient(hunting): authenticate will time out at 821.440124s 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 0x7fdb04131c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 0x7fdb04131c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42330/0 (socket says 172.21.15.104:42330) 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 -- 172.21.15.104:0/324106392 learned_addr learned my addr 172.21.15.104:0/324106392 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:40.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): get_auth_request con 0x7fdb0409f240 auth_method 0 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb040a59d0 0x7fdb0411c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 10 monclient(hunting): get_auth_request con 0x7fdb040a59d0 auth_method 0 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb0409bea0 0x7fdb04140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 10 monclient(hunting): get_auth_request con 0x7fdb0409bea0 auth_method 0 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb02ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 -- 172.21.15.104:0/324106392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb0409bea0 msgr2=0x7fdb04140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb0409bea0 0x7fdb04140ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 -- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb040a59d0 msgr2=0x7fdb0411c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb040a59d0 0x7fdb0411c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.472 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 monclient: found mon.noname-c 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 -- 172.21.15.104:0/324106392 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc009ae0 con 0x7fdb0409f240 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 -- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 msgr2=0x7fdb04131c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafbfff700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 0x7fdb04131c50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400b5e0 tx=0x7fdaf4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 1 -- 172.21.15.104:0/324106392 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf400f720 con 0x7fdb0409f240 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: dump: 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.473 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 5 monclient: authenticate success, global_id 4334 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 1 -- 172.21.15.104:0/324106392 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf400f8a0 con 0x7fdb0409f240 2024-09-18T14:21:40.474 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 1 -- 172.21.15.104:0/324106392 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf400fc60 con 0x7fdb0409f240 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient: _renew_subs 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb04141d10 con 0x7fdb0409f240 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb01ffb700 4 set_mon_vals no callback set 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdafaffd700 10 monclient: dump: 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.475 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient: _renew_subs 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.448+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0411efc0 con 0x7fdb0409f240 2024-09-18T14:21:40.476 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdb03fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdb03fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdb03fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0411fea0 con 0x7fdb0409f240 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdafaffd700 1 -- 172.21.15.104:0/324106392 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fdaf4011610 con 0x7fdb0409f240 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdafaffd700 1 --2- 172.21.15.104:0/324106392 >> v2:172.21.15.174:6812/17603 conn(0x7fdae4035a80 0x7fdae4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdafaffd700 1 -- 172.21.15.104:0/324106392 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7fdaf4047980 con 0x7fdb0409f240 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdb02ffd700 1 --2- 172.21.15.104:0/324106392 >> v2:172.21.15.174:6812/17603 conn(0x7fdae4035a80 0x7fdae4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.452+0000 7fdb02ffd700 10 monclient: get_auth_request con 0x7fdae4035a80 auth_method 0 2024-09-18T14:21:40.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.456+0000 7fdb02ffd700 1 --2- 172.21.15.104:0/324106392 >> v2:172.21.15.174:6812/17603 conn(0x7fdae4035a80 0x7fdae4037f30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009db0 tx=0x7fdaec0091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.460+0000 7fdafaffd700 1 -- 172.21.15.104:0/324106392 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdaf4017a90 con 0x7fdb0409f240 2024-09-18T14:21:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.460+0000 7fdafaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.460+0000 7fdafaffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:40.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.624+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb040633b0 con 0x7fdae4035a80 2024-09-18T14:21:40.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdafaffd700 1 -- 172.21.15.104:0/324106392 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7fdb040633b0 con 0x7fdae4035a80 2024-09-18T14:21:40.639 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 >> v2:172.21.15.174:6812/17603 conn(0x7fdae4035a80 msgr2=0x7fdae4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 --2- 172.21.15.104:0/324106392 >> v2:172.21.15.174:6812/17603 conn(0x7fdae4035a80 0x7fdae4037f30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009db0 tx=0x7fdaec0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 10 monclient: shutdown 2024-09-18T14:21:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 msgr2=0x7fdb04131c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 0x7fdb04131c50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400b5e0 tx=0x7fdaf4009120 comp rx=0 tx=0).stop 2024-09-18T14:21:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 shutdown_connections 2024-09-18T14:21:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 --2- 172.21.15.104:0/324106392 >> v2:172.21.15.174:6812/17603 conn(0x7fdae4035a80 0x7fdae4037f30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb040a59d0 0x7fdb0411c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fdb0409f240 0x7fdb04131c50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 --2- 172.21.15.104:0/324106392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb0409bea0 0x7fdb04140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 >> 172.21.15.104:0/324106392 conn(0x7fdb04067c10 msgr2=0x7fdb040a31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 shutdown_connections 2024-09-18T14:21:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.632+0000 7fdb03fff700 1 -- 172.21.15.104:0/324106392 wait complete. 2024-09-18T14:21:40.644 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:21:40.653 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-18T14:21:40.405865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4778452,"kb_used_data":557418280,"kb_used_omap":444,"kb_used_meta":0,"kb_avail":557417828,"statfs":{"total":575688990720,"available":570795855872,"internally_reserved":462848,"allocated":570796318720,"data_stored":570796318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":455649,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.128588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135537+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:21:35.135537+0000","last_peered":"2024-09-18T14:21:35.135537+0000","last_clean":"2024-09-18T14:21:35.135537+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:21:35.135537+0000","last_undegraded":"2024-09-18T14:21:35.135537+0000","last_fullsized":"2024-09-18T14:21:35.135537+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295699+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:21:34.295699+0000","last_peered":"2024-09-18T14:21:34.295699+0000","last_clean":"2024-09-18T14:21:34.295699+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:21:34.295699+0000","last_undegraded":"2024-09-18T14:21:34.295699+0000","last_fullsized":"2024-09-18T14:21:34.295699+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135401+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:21:35.135401+0000","last_peered":"2024-09-18T14:21:35.135401+0000","last_clean":"2024-09-18T14:21:35.135401+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:21:35.135401+0000","last_undegraded":"2024-09-18T14:21:35.135401+0000","last_fullsized":"2024-09-18T14:21:35.135401+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295840+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:21:34.295840+0000","last_peered":"2024-09-18T14:21:34.295840+0000","last_clean":"2024-09-18T14:21:34.295840+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:21:34.295840+0000","last_undegraded":"2024-09-18T14:21:34.295840+0000","last_fullsized":"2024-09-18T14:21:34.295840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296315+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:21:34.296315+0000","last_peered":"2024-09-18T14:21:34.296315+0000","last_clean":"2024-09-18T14:21:34.296315+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:21:34.296315+0000","last_undegraded":"2024-09-18T14:21:34.296315+0000","last_fullsized":"2024-09-18T14:21:34.296315+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.349394+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:21:34.349394+0000","last_peered":"2024-09-18T14:21:34.349394+0000","last_clean":"2024-09-18T14:21:34.349394+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:21:34.349394+0000","last_undegraded":"2024-09-18T14:21:34.349394+0000","last_fullsized":"2024-09-18T14:21:34.349394+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135845+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:21:35.135845+0000","last_peered":"2024-09-18T14:21:35.135845+0000","last_clean":"2024-09-18T14:21:35.135845+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:21:35.135845+0000","last_undegraded":"2024-09-18T14:21:35.135845+0000","last_fullsized":"2024-09-18T14:21:35.135845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296035+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:21:34.296035+0000","last_peered":"2024-09-18T14:21:34.296035+0000","last_clean":"2024-09-18T14:21:34.296035+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:21:34.296035+0000","last_undegraded":"2024-09-18T14:21:34.296035+0000","last_fullsized":"2024-09-18T14:21:34.296035+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T14:21:33.278313+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:21:33.278313+0000","last_peered":"2024-09-18T14:21:33.278313+0000","last_clean":"2024-09-18T14:21:33.278313+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:21:33.278313+0000","last_undegraded":"2024-09-18T14:21:33.278313+0000","last_fullsized":"2024-09-18T14:21:33.278313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796012,"kb_used_data":92903436,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903368,"statfs":{"total":95948165120,"available":95133048832,"internally_reserved":69632,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58529,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795964,"kb_used_data":92903460,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903416,"statfs":{"total":95948165120,"available":95133097984,"internally_reserved":45056,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48713,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797228,"kb_used_data":92902244,"kb_used_omap":110,"kb_used_meta":0,"kb_avail":92902152,"statfs":{"total":95948165120,"available":95131803648,"internally_reserved":94208,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":113606,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795980,"kb_used_data":92903460,"kb_used_omap":54,"kb_used_meta":0,"kb_avail":92903400,"statfs":{"total":95948165120,"available":95133081600,"internally_reserved":61440,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55928,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796068,"kb_used_data":92903436,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":92903312,"statfs":{"total":95948165120,"available":95132991488,"internally_reserved":126976,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77686,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797200,"kb_used_data":92902244,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92902180,"statfs":{"total":95948165120,"available":95131832320,"internally_reserved":65536,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101187,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]}]}],"pool_statfs":[]}} 2024-09-18T14:21:40.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:40.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient: get_monmap_and_config 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient: build_initial_monmap 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient: monmap: 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:40.755790+0000 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:40.755790+0000 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:40.765 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 1 Processor -- start 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 1 -- start start 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient: init 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa33c0a59d0 0x7fa33c09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient(hunting): picked mon.noname-c con 0x7fa33c0a59d0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa33c09bea0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa33c09f240 0x7fa33c0a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient(hunting): picked mon.noname-a con 0x7fa33c09f240 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c09af10 con 0x7fa33c09bea0 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33c0a1870 con 0x7fa33c09f240 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33c0a5da0 con 0x7fa33c0a59d0 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3421fe700 10 monclient(hunting): authenticate will time out at 821.745375s 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa33c09f240 0x7fa33c0a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa33c09f240 0x7fa33c0a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38330/0 (socket says 172.21.15.104:38330) 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 -- 172.21.15.104:0/2618918644 learned_addr learned my addr 172.21.15.104:0/2618918644 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:40.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): get_auth_request con 0x7fa33c09f240 auth_method 0 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 1 --2- 172.21.15.104:0/2618918644 >> v2:172.21.15.174:3300/0 conn(0x7fa33c0a59d0 0x7fa33c09ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 10 monclient(hunting): get_auth_request con 0x7fa33c0a59d0 auth_method 0 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3411fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 -- 172.21.15.104:0/2618918644 >> v2:172.21.15.174:3300/0 conn(0x7fa33c0a59d0 msgr2=0x7fa33c09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 --2- 172.21.15.104:0/2618918644 >> v2:172.21.15.174:3300/0 conn(0x7fa33c0a59d0 0x7fa33c09ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 -- 172.21.15.104:0/2618918644 >> v1:172.21.15.174:6789/0 conn(0x7fa33c09bea0 legacy=0x7fa33c09fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient: _finish_hunting 0 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 monclient: found mon.noname-a 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.752+0000 7fa3419fd700 1 -- 172.21.15.104:0/2618918644 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c09cae0 con 0x7fa33c09f240 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3419fd700 1 -- 172.21.15.104:0/2618918644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09f240 msgr2=0x7fa33c0a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3419fd700 1 --2- 172.21.15.104:0/2618918644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09f240 0x7fa33c0a03c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa33c09b980 tx=0x7fa334009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d650a90839d3a248 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 1 -- 172.21.15.104:0/2618918644 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33400e070 con 0x7fa33c09f240 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: dump: 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.770 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 5 monclient: authenticate success, global_id 4344 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 1 -- 172.21.15.104:0/2618918644 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33400bc00 con 0x7fa33c09f240 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 4 set_mon_vals no callback set 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 1 -- 172.21.15.104:0/2618918644 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3340092e0 con 0x7fa33c09f240 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa333fff700 10 monclient: dump: 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 10 monclient: shutdown 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 -- 172.21.15.104:0/2618918644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09f240 msgr2=0x7fa33c0a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 --2- 172.21.15.104:0/2618918644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09f240 0x7fa33c0a03c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa33c09b980 tx=0x7fa334009840 comp rx=0 tx=0).stop 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 -- 172.21.15.104:0/2618918644 shutdown_connections 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 --2- 172.21.15.104:0/2618918644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09f240 0x7fa33c0a03c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 --2- 172.21.15.104:0/2618918644 >> v2:172.21.15.174:3300/0 conn(0x7fa33c0a59d0 0x7fa33c09ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 -- 172.21.15.104:0/2618918644 >> 172.21.15.104:0/2618918644 conn(0x7fa33c067c10 msgr2=0x7fa33c0a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 -- 172.21.15.104:0/2618918644 shutdown_connections 2024-09-18T14:21:40.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 -- 172.21.15.104:0/2618918644 wait complete. 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 10 monclient: build_initial_monmap 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 10 monclient: monmap: 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:40.778 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.756+0000 7fa3421fe700 1 Processor -- start 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 1 -- start start 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient: init 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 0x7fa33c131a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): picked mon.noname-a con 0x7fa33c09bea0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa33c09f240 0x7fa33c11b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): picked mon.noname-c con 0x7fa33c09f240 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa33c0a59d0 0x7fa33c11be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): picked mon.noname-b con 0x7fa33c0a59d0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c11eb70 con 0x7fa33c09bea0 2024-09-18T14:21:40.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c11d630 con 0x7fa33c0a59d0 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c11d0d0 con 0x7fa33c09f240 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3421fe700 10 monclient(hunting): authenticate will time out at 821.752823s 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 0x7fa33c131a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa33c09f240 0x7fa33c11b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 0x7fa33c131a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38340/0 (socket says 172.21.15.104:38340) 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 10 monclient(hunting): get_auth_request con 0x7fa33c09f240 auth_method 0 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 -- 172.21.15.104:0/1149727447 learned_addr learned my addr 172.21.15.104:0/1149727447 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): get_auth_request con 0x7fa33c09bea0 auth_method 0 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa33c0a59d0 0x7fa33c11be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 10 monclient(hunting): get_auth_request con 0x7fa33c0a59d0 auth_method 0 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:40.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3409fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3419fd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 -- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa33c09f240 msgr2=0x7fa33c11b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa33c09f240 0x7fa33c11b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 -- 172.21.15.104:0/1149727447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa33c0a59d0 msgr2=0x7fa33c11be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa33c0a59d0 0x7fa33c11be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient: _finish_hunting 0 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 monclient: found mon.noname-a 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:40.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.760+0000 7fa3411fc700 1 -- 172.21.15.104:0/1149727447 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa334009ae0 con 0x7fa33c09bea0 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3411fc700 1 -- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 msgr2=0x7fa33c131a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3411fc700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 0x7fa33c131a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa33800ec80 tx=0x7fa3380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 1 -- 172.21.15.104:0/1149727447 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3380096b0 con 0x7fa33c09bea0 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: dump: 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.783 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: _finish_auth 0 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 5 monclient: authenticate success, global_id 4347 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 10 monclient: _renew_subs 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 1 -- 172.21.15.104:0/1149727447 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33800c9e0 con 0x7fa33c09bea0 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c06f3c0 con 0x7fa33c09bea0 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3337fe700 4 set_mon_vals no callback set 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 10 monclient: _renew_subs 2024-09-18T14:21:40.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c11dc80 con 0x7fa33c09bea0 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 1 -- 172.21.15.104:0/1149727447 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33800cda0 con 0x7fa33c09bea0 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 10 monclient: dump: 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:40.785 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 1 -- 172.21.15.104:0/1149727447 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fa338013680 con 0x7fa33c09bea0 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33c134b90 con 0x7fa33c09bea0 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 1 --2- 172.21.15.104:0/1149727447 >> v2:172.21.15.174:6812/17603 conn(0x7fa320035a80 0x7fa320037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3317fa700 1 -- 172.21.15.104:0/1149727447 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7fa3380481a0 con 0x7fa33c09bea0 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3409fb700 1 --2- 172.21.15.104:0/1149727447 >> v2:172.21.15.174:6812/17603 conn(0x7fa320035a80 0x7fa320037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.764+0000 7fa3409fb700 10 monclient: get_auth_request con 0x7fa320035a80 auth_method 0 2024-09-18T14:21:40.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.768+0000 7fa3409fb700 1 --2- 172.21.15.104:0/1149727447 >> v2:172.21.15.174:6812/17603 conn(0x7fa320035a80 0x7fa320037f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009af0 tx=0x7fa32c009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:40.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.776+0000 7fa3317fa700 1 -- 172.21.15.104:0/1149727447 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa338023ad0 con 0x7fa33c09bea0 2024-09-18T14:21:40.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.776+0000 7fa3317fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:40.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.776+0000 7fa3317fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:40.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.932+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa33c11f090 con 0x7fa320035a80 2024-09-18T14:21:40.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.936+0000 7fa3317fa700 1 -- 172.21.15.104:0/1149727447 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7fa32c00e650 con 0x7fa320035a80 2024-09-18T14:21:40.946 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:21:40.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 >> v2:172.21.15.174:6812/17603 conn(0x7fa320035a80 msgr2=0x7fa320037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 --2- 172.21.15.104:0/1149727447 >> v2:172.21.15.174:6812/17603 conn(0x7fa320035a80 0x7fa320037f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009af0 tx=0x7fa32c009060 comp rx=0 tx=0).stop 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 10 monclient: shutdown 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 msgr2=0x7fa33c131a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 0x7fa33c131a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa33800ec80 tx=0x7fa3380091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 shutdown_connections 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 --2- 172.21.15.104:0/1149727447 >> v2:172.21.15.174:6812/17603 conn(0x7fa320035a80 0x7fa320037f30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa33c0a59d0 0x7fa33c11be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa33c09f240 0x7fa33c11b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 --2- 172.21.15.104:0/1149727447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa33c09bea0 0x7fa33c131a50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:40.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 >> 172.21.15.104:0/1149727447 conn(0x7fa33c067c10 msgr2=0x7fa33c134da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:40.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 shutdown_connections 2024-09-18T14:21:40.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:40.940+0000 7fa3421fe700 1 -- 172.21.15.104:0/1149727447 wait complete. 2024-09-18T14:21:40.949 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:21:40.959 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-18T14:21:40.405865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4778452,"kb_used_data":557418280,"kb_used_omap":444,"kb_used_meta":0,"kb_avail":557417828,"statfs":{"total":575688990720,"available":570795855872,"internally_reserved":462848,"allocated":570796318720,"data_stored":570796318720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":455649,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.128588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135537+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:21:35.135537+0000","last_peered":"2024-09-18T14:21:35.135537+0000","last_clean":"2024-09-18T14:21:35.135537+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:21:35.135537+0000","last_undegraded":"2024-09-18T14:21:35.135537+0000","last_fullsized":"2024-09-18T14:21:35.135537+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295699+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:21:34.295699+0000","last_peered":"2024-09-18T14:21:34.295699+0000","last_clean":"2024-09-18T14:21:34.295699+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:21:34.295699+0000","last_undegraded":"2024-09-18T14:21:34.295699+0000","last_fullsized":"2024-09-18T14:21:34.295699+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135401+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:21:35.135401+0000","last_peered":"2024-09-18T14:21:35.135401+0000","last_clean":"2024-09-18T14:21:35.135401+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:21:35.135401+0000","last_undegraded":"2024-09-18T14:21:35.135401+0000","last_fullsized":"2024-09-18T14:21:35.135401+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.295840+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:21:34.295840+0000","last_peered":"2024-09-18T14:21:34.295840+0000","last_clean":"2024-09-18T14:21:34.295840+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:21:34.295840+0000","last_undegraded":"2024-09-18T14:21:34.295840+0000","last_fullsized":"2024-09-18T14:21:34.295840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296315+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:21:34.296315+0000","last_peered":"2024-09-18T14:21:34.296315+0000","last_clean":"2024-09-18T14:21:34.296315+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:21:34.296315+0000","last_undegraded":"2024-09-18T14:21:34.296315+0000","last_fullsized":"2024-09-18T14:21:34.296315+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.349394+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:21:34.349394+0000","last_peered":"2024-09-18T14:21:34.349394+0000","last_clean":"2024-09-18T14:21:34.349394+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:21:34.349394+0000","last_undegraded":"2024-09-18T14:21:34.349394+0000","last_fullsized":"2024-09-18T14:21:34.349394+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:35.135845+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:21:35.135845+0000","last_peered":"2024-09-18T14:21:35.135845+0000","last_clean":"2024-09-18T14:21:35.135845+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:21:35.135845+0000","last_undegraded":"2024-09-18T14:21:35.135845+0000","last_fullsized":"2024-09-18T14:21:35.135845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T14:21:34.296035+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:21:34.296035+0000","last_peered":"2024-09-18T14:21:34.296035+0000","last_clean":"2024-09-18T14:21:34.296035+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:21:34.296035+0000","last_undegraded":"2024-09-18T14:21:34.296035+0000","last_fullsized":"2024-09-18T14:21:34.296035+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T14:21:33.278313+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:21:33.278313+0000","last_peered":"2024-09-18T14:21:33.278313+0000","last_clean":"2024-09-18T14:21:33.278313+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:21:33.278313+0000","last_undegraded":"2024-09-18T14:21:33.278313+0000","last_fullsized":"2024-09-18T14:21:33.278313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796012,"kb_used_data":92903436,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903368,"statfs":{"total":95948165120,"available":95133048832,"internally_reserved":69632,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58529,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795964,"kb_used_data":92903460,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903416,"statfs":{"total":95948165120,"available":95133097984,"internally_reserved":45056,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48713,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797228,"kb_used_data":92902244,"kb_used_omap":110,"kb_used_meta":0,"kb_avail":92902152,"statfs":{"total":95948165120,"available":95131803648,"internally_reserved":94208,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":113606,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795980,"kb_used_data":92903460,"kb_used_omap":54,"kb_used_meta":0,"kb_avail":92903400,"statfs":{"total":95948165120,"available":95133081600,"internally_reserved":61440,"allocated":95133143040,"data_stored":95133143040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55928,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796068,"kb_used_data":92903436,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":92903312,"statfs":{"total":95948165120,"available":95132991488,"internally_reserved":126976,"allocated":95133118464,"data_stored":95133118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77686,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797200,"kb_used_data":92902244,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92902180,"statfs":{"total":95948165120,"available":95131832320,"internally_reserved":65536,"allocated":95131897856,"data_stored":95131897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101187,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]}]}],"pool_statfs":[]}} 2024-09-18T14:21:40.960 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-18T14:21:40.961 DEBUG:tasks.mon_thrash:Yielding 2024-09-18T14:21:40.961 INFO:teuthology.run_tasks:Running task exec... 2024-09-18T14:21:40.969 INFO:teuthology.task.exec:Executing custom commands... 2024-09-18T14:21:40.969 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:21:40.970 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:21:40.970 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-18T14:21:40.970 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726669300, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-18T14:21:40.970 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T14:21:40.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:21:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.060+0000 7fe6e9144700 10 monclient: get_monmap_and_config 2024-09-18T14:21:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.060+0000 7fe6e9144700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.060+0000 7fe6e9144700 10 monclient: monmap: 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:41.066673+0000 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:41.066673+0000 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:41.070 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 1 Processor -- start 2024-09-18T14:21:41.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 1 -- start start 2024-09-18T14:21:41.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient: init 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6e412e4b0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6e406e950 0x7fe6e406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient(hunting): picked mon.noname-a con 0x7fe6e406e950 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6e4133610 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e406ce60 con 0x7fe6e412e4b0 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e406be30 con 0x7fe6e4133610 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e409d8b0 con 0x7fe6e406e950 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6e406e950 0x7fe6e406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e9144700 10 monclient(hunting): authenticate will time out at 822.057997s 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e2d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe6e412e4b0 0x7fe6e412e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:50110/0 (socket says 172.21.15.104:50110) 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6e406e950 0x7fe6e406ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38354/0 (socket says 172.21.15.104:38354) 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e2d9d700 1 -- 172.21.15.104:0/1650795235 learned_addr learned my addr 172.21.15.104:0/1650795235 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): get_auth_request con 0x7fe6e406e950 auth_method 0 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 1 -- 172.21.15.104:0/1650795235 >> v1:172.21.15.174:6789/0 conn(0x7fe6e4133610 legacy=0x7fe6e41359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 1 -- 172.21.15.104:0/1650795235 >> v1:172.21.15.104:6789/0 conn(0x7fe6e412e4b0 legacy=0x7fe6e412e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 1 monclient: found mon.noname-a 2024-09-18T14:21:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.064+0000 7fe6e259c700 1 -- 172.21.15.104:0/1650795235 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e406cb80 con 0x7fe6e406e950 2024-09-18T14:21:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 1 -- 172.21.15.104:0/1650795235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 msgr2=0x7fe6e406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 1 --2- 172.21.15.104:0/1650795235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e406ed50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800aa90 tx=0x7fe6d800a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf163e3e1e4acac4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 1 -- 172.21.15.104:0/1650795235 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d800cd30 con 0x7fe6e406e950 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: dump: 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 5 monclient: authenticate success, global_id 4362 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 1 -- 172.21.15.104:0/1650795235 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d800ce90 con 0x7fe6e406e950 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 4 set_mon_vals no callback set 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 1 -- 172.21.15.104:0/1650795235 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8018640 con 0x7fe6e406e950 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e1d9b700 10 monclient: dump: 2024-09-18T14:21:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient: shutdown 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- 172.21.15.104:0/1650795235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 msgr2=0x7fe6e406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 --2- 172.21.15.104:0/1650795235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e406ed50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800aa90 tx=0x7fe6d800a330 comp rx=0 tx=0).stop 2024-09-18T14:21:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- 172.21.15.104:0/1650795235 shutdown_connections 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 --2- 172.21.15.104:0/1650795235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e406ed50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- 172.21.15.104:0/1650795235 >> 172.21.15.104:0/1650795235 conn(0x7fe6e40682c0 msgr2=0x7fe6e40a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- 172.21.15.104:0/1650795235 shutdown_connections 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- 172.21.15.104:0/1650795235 wait complete. 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient: monmap: 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 Processor -- start 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- start start 2024-09-18T14:21:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient: init 2024-09-18T14:21:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e409b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): picked mon.noname-a con 0x7fe6e406e950 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6e412e4b0 0x7fe6e41180b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): picked mon.noname-b con 0x7fe6e412e4b0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe6e4133610 0x7fe6e41185f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): picked mon.noname-c con 0x7fe6e4133610 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e4119e90 con 0x7fe6e406e950 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e409be00 con 0x7fe6e412e4b0 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e406e720 con 0x7fe6e4133610 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e9144700 10 monclient(hunting): authenticate will time out at 822.063447s 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e409b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e409b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38366/0 (socket says 172.21.15.104:38366) 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 1 -- 172.21.15.104:0/184942931 learned_addr learned my addr 172.21.15.104:0/184942931 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 10 monclient(hunting): get_auth_request con 0x7fe6e406e950 auth_method 0 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe6e4133610 0x7fe6e41185f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 10 monclient(hunting): get_auth_request con 0x7fe6e4133610 auth_method 0 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6e412e4b0 0x7fe6e41180b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 10 monclient(hunting): get_auth_request con 0x7fe6e412e4b0 auth_method 0 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.068+0000 7fe6e259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 -- 172.21.15.104:0/184942931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6e412e4b0 msgr2=0x7fe6e41180b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6e412e4b0 0x7fe6e41180b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 -- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 msgr2=0x7fe6e409b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e409b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 monclient: found mon.noname-c 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 -- 172.21.15.104:0/184942931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d800a5d0 con 0x7fe6e4133610 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e2d9d700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e409b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 -- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe6e4133610 msgr2=0x7fe6e41185f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e359e700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe6e4133610 0x7fe6e41185f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00c840 tx=0x7fe6dc002620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 1 -- 172.21.15.104:0/184942931 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc012840 con 0x7fe6e4133610 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: dump: 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 1 -- 172.21.15.104:0/184942931 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc0129c0 con 0x7fe6e4133610 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e9144700 5 monclient: authenticate success, global_id 4349 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e9144700 10 monclient: _renew_subs 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e9144700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e411adf0 con 0x7fe6e4133610 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 1 -- 172.21.15.104:0/184942931 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc012d80 con 0x7fe6e4133610 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6caffd700 10 monclient: dump: 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e159a700 4 set_mon_vals no callback set 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e9144700 10 monclient: _renew_subs 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e9144700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.072+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e412de40 con 0x7fe6e4133610 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6caffd700 1 -- 172.21.15.104:0/184942931 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fe6dc011590 con 0x7fe6e4133610 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6e9144700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6e9144700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6e9144700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e411b290 con 0x7fe6e4133610 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6caffd700 1 --2- 172.21.15.104:0/184942931 >> v2:172.21.15.174:6812/17603 conn(0x7fe6d0035ad0 0x7fe6d0037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6caffd700 1 -- 172.21.15.104:0/184942931 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7fe6dc047af0 con 0x7fe6e4133610 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6e2d9d700 1 --2- 172.21.15.104:0/184942931 >> v2:172.21.15.174:6812/17603 conn(0x7fe6d0035ad0 0x7fe6d0037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6e2d9d700 10 monclient: get_auth_request con 0x7fe6d0035ad0 auth_method 0 2024-09-18T14:21:41.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.076+0000 7fe6e2d9d700 1 --2- 172.21.15.104:0/184942931 >> v2:172.21.15.174:6812/17603 conn(0x7fe6d0035ad0 0x7fe6d0037f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40052e0 tx=0x7fe6d4004a80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.080+0000 7fe6caffd700 1 -- 172.21.15.104:0/184942931 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe6dc0198e0 con 0x7fe6e4133610 2024-09-18T14:21:41.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.080+0000 7fe6caffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.080+0000 7fe6caffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:41.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:21:41.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:41.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe6e40636d0 con 0x7fe6e4133610 2024-09-18T14:21:41.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6caffd700 1 -- 172.21.15.104:0/184942931 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fe6dc016030 con 0x7fe6e4133610 2024-09-18T14:21:41.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6caffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:41.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6caffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:41.315 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:21:41.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 >> v2:172.21.15.174:6812/17603 conn(0x7fe6d0035ad0 msgr2=0x7fe6d0037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 --2- 172.21.15.104:0/184942931 >> v2:172.21.15.174:6812/17603 conn(0x7fe6d0035ad0 0x7fe6d0037f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40052e0 tx=0x7fe6d4004a80 comp rx=0 tx=0).stop 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 10 monclient: shutdown 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe6e4133610 msgr2=0x7fe6e41185f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe6e4133610 0x7fe6e41185f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00c840 tx=0x7fe6dc002620 comp rx=0 tx=0).stop 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 shutdown_connections 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 --2- 172.21.15.104:0/184942931 >> v2:172.21.15.174:6812/17603 conn(0x7fe6d0035ad0 0x7fe6d0037f80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe6e4133610 0x7fe6e41185f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6e412e4b0 0x7fe6e41180b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 --2- 172.21.15.104:0/184942931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e406e950 0x7fe6e409b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 >> 172.21.15.104:0/184942931 conn(0x7fe6e40682c0 msgr2=0x7fe6e40638e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 shutdown_connections 2024-09-18T14:21:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.308+0000 7fe6e9144700 1 -- 172.21.15.104:0/184942931 wait complete. 2024-09-18T14:21:41.329 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T14:21:41.329 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T14:21:41.329 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T14:21:41.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:21:41.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.428+0000 7feac9183700 10 monclient: get_monmap_and_config 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.428+0000 7feac9183700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.428+0000 7feac9183700 10 monclient: monmap: 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:41.434960+0000 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:41.434960+0000 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.438 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.439 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:41.439 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:41.439 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:41.439 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:41.439 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:41.439 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.428+0000 7feac9183700 1 Processor -- start 2024-09-18T14:21:41.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.428+0000 7feac9183700 1 -- start start 2024-09-18T14:21:41.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.428+0000 7feac9183700 10 monclient: init 2024-09-18T14:21:41.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T14:21:41.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac4097c10 0x7feac4096300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient(hunting): picked mon.noname-b con 0x7feac4097c10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7feac409bbe0 0x7feac409bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient(hunting): picked mon.noname-c con 0x7feac409bbe0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feac409c4f0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac4095920 con 0x7feac409c4f0 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac4097fe0 con 0x7feac409bbe0 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac409cf10 con 0x7feac4097c10 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient(hunting): authenticate will time out at 822.424503s 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac4097c10 0x7feac4096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac8982700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feac409c4f0 0x7feac4136390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:50118/0 (socket says 172.21.15.104:50118) 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac8982700 1 -- 172.21.15.104:0/1409281207 learned_addr learned my addr 172.21.15.104:0/1409281207 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): get_auth_request con 0x7feac4097c10 auth_method 0 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 1 --2- 172.21.15.104:0/1409281207 >> v2:172.21.15.174:3300/0 conn(0x7feac409bbe0 0x7feac409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 10 monclient(hunting): get_auth_request con 0x7feac409bbe0 auth_method 0 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 -- 172.21.15.104:0/1409281207 >> v2:172.21.15.174:3300/0 conn(0x7feac409bbe0 msgr2=0x7feac409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 --2- 172.21.15.104:0/1409281207 >> v2:172.21.15.174:3300/0 conn(0x7feac409bbe0 0x7feac409bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 -- 172.21.15.104:0/1409281207 >> v1:172.21.15.104:6789/0 conn(0x7feac409c4f0 legacy=0x7feac4136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 monclient: found mon.noname-b 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 -- 172.21.15.104:0/1409281207 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac4095640 con 0x7feac4097c10 2024-09-18T14:21:41.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 -- 172.21.15.104:0/1409281207 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac4097c10 msgr2=0x7feac4096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac3fff700 1 --2- 172.21.15.104:0/1409281207 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac4097c10 0x7feac4096300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feab40098e0 tx=0x7feab40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4bc7840b006cc5f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 1 -- 172.21.15.104:0/1409281207 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab400b850 con 0x7feac4097c10 2024-09-18T14:21:41.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:41.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: dump: 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 5 monclient: authenticate success, global_id 4352 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 1 -- 172.21.15.104:0/1409281207 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab400b9b0 con 0x7feac4097c10 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac9183700 4 set_mon_vals no callback set 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 1 -- 172.21.15.104:0/1409281207 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab400bd50 con 0x7feac4097c10 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.432+0000 7feac2ffd700 10 monclient: dump: 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient: shutdown 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- 172.21.15.104:0/1409281207 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac4097c10 msgr2=0x7feac4096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 --2- 172.21.15.104:0/1409281207 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac4097c10 0x7feac4096300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feab40098e0 tx=0x7feab40091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- 172.21.15.104:0/1409281207 shutdown_connections 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 --2- 172.21.15.104:0/1409281207 >> v2:172.21.15.174:3300/0 conn(0x7feac409bbe0 0x7feac409bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 --2- 172.21.15.104:0/1409281207 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac4097c10 0x7feac4096300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- 172.21.15.104:0/1409281207 >> 172.21.15.104:0/1409281207 conn(0x7feac40a11f0 msgr2=0x7feac40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- 172.21.15.104:0/1409281207 shutdown_connections 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- 172.21.15.104:0/1409281207 wait complete. 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient: monmap: 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 Processor -- start 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- start start 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient: init 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 0x7feac413f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): picked mon.noname-a con 0x7feac4097c10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac409bbe0 0x7feac413f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): picked mon.noname-c con 0x7feac409bbe0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feac409c4f0 0x7feac413fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): picked mon.noname-b con 0x7feac409c4f0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac408e270 con 0x7feac4097c10 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac409d440 con 0x7feac409c4f0 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac408da30 con 0x7feac409bbe0 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac9183700 10 monclient(hunting): authenticate will time out at 822.430631s 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 0x7feac413f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 0x7feac413f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38370/0 (socket says 172.21.15.104:38370) 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 -- 172.21.15.104:0/4225023062 learned_addr learned my addr 172.21.15.104:0/4225023062 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): get_auth_request con 0x7feac4097c10 auth_method 0 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac409bbe0 0x7feac413f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 10 monclient(hunting): get_auth_request con 0x7feac409bbe0 auth_method 0 2024-09-18T14:21:41.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac8982700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feac409c4f0 0x7feac413fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-09-18T14:21:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac409bbe0 msgr2=0x7feac413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7feac409bbe0 0x7feac413f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feac409c4f0 msgr2=0x7feac413fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feac409c4f0 0x7feac413fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 monclient: found mon.noname-a 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac3fff700 1 -- 172.21.15.104:0/4225023062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4009420 con 0x7feac4097c10 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac8982700 10 monclient: get_auth_request con 0x7feac409c4f0 auth_method 0 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.436+0000 7feac8982700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feac409c4f0 0x7feac413fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac3fff700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 msgr2=0x7feac413f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac3fff700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 0x7feac413f450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feab40093f0 tx=0x7feab4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4000d10 con 0x7feac4097c10 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: dump: 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.458 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 5 monclient: authenticate success, global_id 4368 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab4011e20 con 0x7feac4097c10 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 10 monclient: _renew_subs 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac408d520 con 0x7feac4097c10 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac27fc700 4 set_mon_vals no callback set 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 10 monclient: _renew_subs 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac406e6f0 con 0x7feac4097c10 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab401b8d0 con 0x7feac4097c10 2024-09-18T14:21:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 10 monclient: dump: 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7feab401bc40 con 0x7feac4097c10 2024-09-18T14:21:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 1 --2- 172.21.15.104:0/4225023062 >> v2:172.21.15.174:6812/17603 conn(0x7feaa8035a80 0x7feaa8037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7feab4049a10 con 0x7feac4097c10 2024-09-18T14:21:41.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 0x7feac405e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40686b0 con 0x7feac408cb50 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac37fe700 1 --2- 172.21.15.104:0/4225023062 >> v2:172.21.15.174:6812/17603 conn(0x7feaa8035a80 0x7feaa8037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac37fe700 10 monclient: get_auth_request con 0x7feaa8035a80 auth_method 0 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac8982700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 0x7feac405e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac8982700 10 monclient: get_auth_request con 0x7feac408cb50 auth_method 0 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac8982700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac8982700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.440+0000 7feac8982700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac8982700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac8982700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac8982700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac8982700 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac8982700 1 -- 172.21.15.104:0/4225023062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feac4063450 con 0x7feac408cb50 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac37fe700 1 --2- 172.21.15.104:0/4225023062 >> v2:172.21.15.174:6812/17603 conn(0x7feaa8035a80 0x7feaa8037f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feaac009db0 tx=0x7feaac0091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac8982700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 msgr2=0x7feac405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feac8982700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 0x7feac405e890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feabc00dbc0 tx=0x7feabc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc00bc10 con 0x7feac408cb50 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 10 monclient: dump: 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.464 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7feabc00bd70 con 0x7feac408cb50 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 msgr2=0x7feac405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 0x7feac405e890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feabc00dbc0 tx=0x7feabc009040 comp rx=0 tx=0).stop 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 msgr2=0x7feac405e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.444+0000 7feabbfff700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac408cb50 0x7feac405e890 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac9183700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:21:41.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac9183700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 0x7feac4029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac9183700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac408f1e0 con 0x7feac405edd0 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 0x7feac4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient: get_auth_request con 0x7feac405edd0 auth_method 0 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-09-18T14:21:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.456+0000 7feac8982700 1 -- 172.21.15.104:0/4225023062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feac4088740 con 0x7feac405edd0 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac8982700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 msgr2=0x7feac4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac8982700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 0x7feac4029d00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feabc00dbc0 tx=0x7feabc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc012b70 con 0x7feac405edd0 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 10 monclient: dump: 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.468 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.469 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.469 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.469 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.469 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.469 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.469 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.469 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7feabc00ba00 con 0x7feac405edd0 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 msgr2=0x7feac4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 0x7feac4029d00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feabc00dbc0 tx=0x7feabc019000 comp rx=0 tx=0).stop 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 msgr2=0x7feac4029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feabbfff700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac405edd0 0x7feac4029d00 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feabc00dbc0 tx=0x7feabc019000 comp rx=0 tx=0).stop 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:21:41.470 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 8, 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 22, 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:21:41.471 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.472 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:21:41.473 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.474 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:21:41.475 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.476 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:21:41.477 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.479 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.480 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.481 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.482 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:21:41.482 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.482 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.482 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:21:41.482 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:21:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 >> v2:172.21.15.174:6812/17603 conn(0x7feaa8035a80 msgr2=0x7feaa8037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 --2- 172.21.15.104:0/4225023062 >> v2:172.21.15.174:6812/17603 conn(0x7feaa8035a80 0x7feaa8037f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feaac009db0 tx=0x7feaac0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 10 monclient: shutdown 2024-09-18T14:21:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac8982700 1 -- 172.21.15.104:0/4225023062 reap_dead start 2024-09-18T14:21:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 msgr2=0x7feac413f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 0x7feac413f450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feab40093f0 tx=0x7feab4016000 comp rx=0 tx=0).stop 2024-09-18T14:21:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 shutdown_connections 2024-09-18T14:21:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 --2- 172.21.15.104:0/4225023062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feac4097c10 0x7feac413f450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 >> 172.21.15.104:0/4225023062 conn(0x7feac40a11f0 msgr2=0x7feac409f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 shutdown_connections 2024-09-18T14:21:41.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.460+0000 7feac9183700 1 -- 172.21.15.104:0/4225023062 wait complete. 2024-09-18T14:21:41.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:21:41.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T14:21:41.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T14:21:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient: get_monmap_and_config 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient: monmap: 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:41.585830+0000 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:41.585830+0000 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:41.590 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 1 Processor -- start 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 1 -- start start 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient: init 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe0c4063f20 0x7fe0c406b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient(hunting): picked mon.noname-a con 0x7fe0c4063f20 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0c406bc40 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe0c406c040 0x7fe0c4136760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient(hunting): picked mon.noname-c con 0x7fe0c406c040 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40642f0 con 0x7fe0c406bc40 2024-09-18T14:21:41.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c409d8d0 con 0x7fe0c4063f20 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c409db10 con 0x7fe0c406c040 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c8b09700 10 monclient(hunting): authenticate will time out at 822.575771s 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe0c4063f20 0x7fe0c406b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe0c4063f20 0x7fe0c406b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38382/0 (socket says 172.21.15.104:38382) 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4147679202 learned_addr learned my addr 172.21.15.104:0/4147679202 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c37fe700 10 monclient(hunting): get_auth_request con 0x7fe0c4063f20 auth_method 0 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.580+0000 7fe0c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c3fff700 1 --2- 172.21.15.104:0/4147679202 >> v2:172.21.15.174:3300/0 conn(0x7fe0c406c040 0x7fe0c4136760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c3fff700 10 monclient(hunting): get_auth_request con 0x7fe0c406c040 auth_method 0 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4147679202 >> v2:172.21.15.174:3300/0 conn(0x7fe0c406c040 msgr2=0x7fe0c4136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 1 --2- 172.21.15.104:0/4147679202 >> v2:172.21.15.174:3300/0 conn(0x7fe0c406c040 0x7fe0c4136760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4147679202 >> v1:172.21.15.104:6789/0 conn(0x7fe0c406bc40 legacy=0x7fe0c4133030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 1 monclient: found mon.noname-a 2024-09-18T14:21:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4147679202 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c406cfe0 con 0x7fe0c4063f20 2024-09-18T14:21:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4147679202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 msgr2=0x7fe0c406b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c37fe700 1 --2- 172.21.15.104:0/4147679202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c406b700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0c406d3c0 tx=0x7fe0b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cccf41754284d24 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 1 -- 172.21.15.104:0/4147679202 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b400ba70 con 0x7fe0c4063f20 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: dump: 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 1 -- 172.21.15.104:0/4147679202 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b400bbd0 con 0x7fe0c4063f20 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c8b09700 5 monclient: authenticate success, global_id 4377 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c8b09700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c8b09700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 1 -- 172.21.15.104:0/4147679202 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4011420 con 0x7fe0c4063f20 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c8b09700 4 set_mon_vals no callback set 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.584+0000 7fe0c27fc700 10 monclient: dump: 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient: shutdown 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4147679202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 msgr2=0x7fe0c406b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 --2- 172.21.15.104:0/4147679202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c406b700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0c406d3c0 tx=0x7fe0b4009250 comp rx=0 tx=0).stop 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4147679202 shutdown_connections 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 --2- 172.21.15.104:0/4147679202 >> v2:172.21.15.174:3300/0 conn(0x7fe0c406c040 0x7fe0c4136760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 --2- 172.21.15.104:0/4147679202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c406b700 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4147679202 >> 172.21.15.104:0/4147679202 conn(0x7fe0c40682c0 msgr2=0x7fe0c40a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4147679202 shutdown_connections 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4147679202 wait complete. 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient: monmap: 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 Processor -- start 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- start start 2024-09-18T14:21:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient: init 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c41330e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): picked mon.noname-a con 0x7fe0c4063f20 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c406bc40 0x7fe0c411cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): picked mon.noname-c con 0x7fe0c406bc40 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0c406c040 0x7fe0c411d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): picked mon.noname-b con 0x7fe0c406c040 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4120200 con 0x7fe0c4063f20 2024-09-18T14:21:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4120830 con 0x7fe0c406c040 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c411ebe0 con 0x7fe0c406bc40 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c8b09700 10 monclient(hunting): authenticate will time out at 822.583478s 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c41330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c2ffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c406bc40 0x7fe0c411cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c41330e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38394/0 (socket says 172.21.15.104:38394) 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4274312014 learned_addr learned my addr 172.21.15.104:0/4274312014 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 10 monclient(hunting): get_auth_request con 0x7fe0c4063f20 auth_method 0 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c2ffd700 10 monclient(hunting): get_auth_request con 0x7fe0c406bc40 auth_method 0 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c3fff700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0c406c040 0x7fe0c411d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c3fff700 10 monclient(hunting): get_auth_request con 0x7fe0c406c040 auth_method 0 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.588+0000 7fe0c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c406bc40 msgr2=0x7fe0c411cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c406bc40 0x7fe0c411cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0c406c040 msgr2=0x7fe0c411d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0c406c040 0x7fe0c411d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 monclient: found mon.noname-a 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4274312014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b40094f0 con 0x7fe0c4063f20 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 msgr2=0x7fe0c41330e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c37fe700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c41330e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400b3c0 tx=0x7fe0b4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b400ba70 con 0x7fe0c4063f20 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: dump: 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.612 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c8b09700 5 monclient: authenticate success, global_id 4380 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b4020070 con 0x7fe0c4063f20 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b401c680 con 0x7fe0c4063f20 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c8b09700 10 monclient: _renew_subs 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c8b09700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c406c710 con 0x7fe0c4063f20 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c1ffb700 4 set_mon_vals no callback set 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0bb7fe700 10 monclient: dump: 2024-09-18T14:21:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c8b09700 10 monclient: _renew_subs 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c8b09700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.592+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c411e6d0 con 0x7fe0c4063f20 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c8b09700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c8b09700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 0x7fe0c4029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c8b09700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4068e40 con 0x7fe0c411df90 2024-09-18T14:21:41.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 0x7fe0c4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient: get_auth_request con 0x7fe0c411df90 auth_method 0 2024-09-18T14:21:41.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 1 -- 172.21.15.104:0/4274312014 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0c4119600 con 0x7fe0c411df90 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fe0b4022030 con 0x7fe0c4063f20 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0bb7fe700 1 --2- 172.21.15.104:0/4274312014 >> v2:172.21.15.174:6812/17603 conn(0x7fe0a4035a80 0x7fe0a4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fe0b4049d30 con 0x7fe0c4063f20 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c3fff700 1 --2- 172.21.15.104:0/4274312014 >> v2:172.21.15.174:6812/17603 conn(0x7fe0a4035a80 0x7fe0a4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c3fff700 10 monclient: get_auth_request con 0x7fe0a4035a80 auth_method 0 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 msgr2=0x7fe0c4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0c2ffd700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 0x7fe0c4029d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00f7b0 tx=0x7fe0ac007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac014890 con 0x7fe0c411df90 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0bb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.596+0000 7fe0bb7fe700 10 monclient: dump: 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.617 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe0ac0149f0 con 0x7fe0c411df90 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0bb7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0bb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 msgr2=0x7fe0c4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0bb7fe700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 0x7fe0c4029d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00f7b0 tx=0x7fe0ac007ea0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 msgr2=0x7fe0c4029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0bb7fe700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c411df90 0x7fe0c4029d00 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00f7b0 tx=0x7fe0ac007ea0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.600+0000 7fe0c3fff700 1 --2- 172.21.15.104:0/4274312014 >> v2:172.21.15.174:6812/17603 conn(0x7fe0a4035a80 0x7fe0a4037f30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4120e50 tx=0x7fe0bc009850 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c8b09700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c8b09700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 0x7fe0c402a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c8b09700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4063450 con 0x7fe0c4133620 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 0x7fe0c402a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient: get_auth_request con 0x7fe0c4133620 auth_method 0 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.608+0000 7fe0c2ffd700 1 -- 172.21.15.104:0/4274312014 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe0c4120440 con 0x7fe0c4133620 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c2ffd700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 msgr2=0x7fe0c402a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c2ffd700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 0x7fe0c402a240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac003480 tx=0x7fe0ac019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac021620 con 0x7fe0c4133620 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 10 monclient: dump: 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.621 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fe0ac021780 con 0x7fe0c4133620 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 msgr2=0x7fe0c402a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 0x7fe0c402a240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac003480 tx=0x7fe0ac019000 comp rx=0 tx=0).stop 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 msgr2=0x7fe0c402a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0bb7fe700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0c4133620 0x7fe0c402a240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 8, 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:21:41.623 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 22, 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.624 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:21:41.626 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.627 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:21:41.628 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.629 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:21:41.630 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.633 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:21:41.634 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:21:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 >> v2:172.21.15.174:6812/17603 conn(0x7fe0a4035a80 msgr2=0x7fe0a4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 --2- 172.21.15.104:0/4274312014 >> v2:172.21.15.174:6812/17603 conn(0x7fe0a4035a80 0x7fe0a4037f30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4120e50 tx=0x7fe0bc009850 comp rx=0 tx=0).stop 2024-09-18T14:21:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 10 monclient: shutdown 2024-09-18T14:21:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 msgr2=0x7fe0c41330e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 --2- 172.21.15.104:0/4274312014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c4063f20 0x7fe0c41330e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400b3c0 tx=0x7fe0b4016000 comp rx=0 tx=0).stop 2024-09-18T14:21:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c3fff700 1 -- 172.21.15.104:0/4274312014 reap_dead start 2024-09-18T14:21:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c3fff700 1 -- 172.21.15.104:0/4274312014 reap_dead start 2024-09-18T14:21:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 shutdown_connections 2024-09-18T14:21:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 >> 172.21.15.104:0/4274312014 conn(0x7fe0c40682c0 msgr2=0x7fe0c4095c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 shutdown_connections 2024-09-18T14:21:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.612+0000 7fe0c8b09700 1 -- 172.21.15.104:0/4274312014 wait complete. 2024-09-18T14:21:41.638 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:21:41.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T14:21:41.638 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:21:41.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.736+0000 7f45fbcf8700 10 monclient: get_monmap_and_config 2024-09-18T14:21:41.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.736+0000 7f45fbcf8700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.736+0000 7f45fbcf8700 10 monclient: monmap: 2024-09-18T14:21:41.746 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.746 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:41.743537+0000 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:41.743537+0000 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:41.747 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 1 Processor -- start 2024-09-18T14:21:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 1 -- start start 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient: init 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f409d620 0x7f45f412eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient(hunting): picked mon.noname-b con 0x7f45f409d620 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45f412f080 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f45f412f450 0x7f45f40961f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient(hunting): picked mon.noname-c con 0x7f45f412f450 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f40990f0 con 0x7f45f412f080 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f409d9f0 con 0x7f45f412f450 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4096e10 con 0x7f45f409d620 2024-09-18T14:21:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fbcf8700 10 monclient(hunting): authenticate will time out at 822.733656s 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9293700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f45f412f080 0x7f45f4095ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:50146/0 (socket says 172.21.15.104:50146) 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9293700 1 -- 172.21.15.104:0/3425882226 learned_addr learned my addr 172.21.15.104:0/3425882226 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 1 --2- 172.21.15.104:0/3425882226 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f409d620 0x7f45f412eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 1 --2- 172.21.15.104:0/3425882226 >> v2:172.21.15.174:3300/0 conn(0x7f45f412f450 0x7f45f40961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 10 monclient(hunting): get_auth_request con 0x7f45f409d620 auth_method 0 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): get_auth_request con 0x7f45f412f450 auth_method 0 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45f9a94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 1 -- 172.21.15.104:0/3425882226 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f409d620 msgr2=0x7f45f412eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 1 --2- 172.21.15.104:0/3425882226 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f409d620 0x7f45f412eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 1 -- 172.21.15.104:0/3425882226 >> v1:172.21.15.104:6789/0 conn(0x7f45f412f080 legacy=0x7f45f4095ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 1 monclient: found mon.noname-c 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.740+0000 7f45fa295700 1 -- 172.21.15.104:0/3425882226 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f409c200 con 0x7f45f412f450 2024-09-18T14:21:41.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fa295700 1 -- 172.21.15.104:0/3425882226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f450 msgr2=0x7f45f40961f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fa295700 1 --2- 172.21.15.104:0/3425882226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f450 0x7f45f40961f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0098e0 tx=0x7f45ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74713d2b9b0ab0c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 1 -- 172.21.15.104:0/3425882226 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec00b850 con 0x7f45f412f450 2024-09-18T14:21:41.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: dump: 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 1 -- 172.21.15.104:0/3425882226 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec00b9b0 con 0x7f45f412f450 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 1 -- 172.21.15.104:0/3425882226 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec00bd50 con 0x7f45f412f450 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 5 monclient: authenticate success, global_id 4378 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 4 set_mon_vals no callback set 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45f8a92700 10 monclient: dump: 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 10 monclient: shutdown 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 -- 172.21.15.104:0/3425882226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f450 msgr2=0x7f45f40961f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/3425882226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f450 0x7f45f40961f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0098e0 tx=0x7f45ec0091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 -- 172.21.15.104:0/3425882226 shutdown_connections 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/3425882226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f450 0x7f45f40961f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/3425882226 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f409d620 0x7f45f412eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 -- 172.21.15.104:0/3425882226 >> 172.21.15.104:0/3425882226 conn(0x7f45f40a1290 msgr2=0x7f45f40a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 -- 172.21.15.104:0/3425882226 shutdown_connections 2024-09-18T14:21:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 1 -- 172.21.15.104:0/3425882226 wait complete. 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.744+0000 7f45fbcf8700 10 monclient: monmap: 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 Processor -- start 2024-09-18T14:21:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 -- start start 2024-09-18T14:21:41.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient: init 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45f409d620 0x7f45f409c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): picked mon.noname-a con 0x7f45f409d620 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f080 0x7f45f412e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): picked mon.noname-b con 0x7f45f412f080 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f412f450 0x7f45f411cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): picked mon.noname-c con 0x7f45f412f450 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f409d1b0 con 0x7f45f409d620 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f409a670 con 0x7f45f412f080 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f406f370 con 0x7f45f412f450 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fbcf8700 10 monclient(hunting): authenticate will time out at 822.741031s 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45f409d620 0x7f45f409c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f412f450 0x7f45f411cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45f409d620 0x7f45f409c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38406/0 (socket says 172.21.15.104:38406) 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): get_auth_request con 0x7f45f412f450 auth_method 0 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 1 -- 172.21.15.104:0/2483664379 learned_addr learned my addr 172.21.15.104:0/2483664379 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:41.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 10 monclient(hunting): get_auth_request con 0x7f45f409d620 auth_method 0 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f080 0x7f45f412e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 10 monclient(hunting): get_auth_request con 0x7f45f412f080 auth_method 0 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f080 msgr2=0x7f45f412e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412f080 0x7f45f412e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45f409d620 msgr2=0x7f45f409c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45f409d620 0x7f45f409c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 1 monclient: found mon.noname-c 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45fa295700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec009420 con 0x7f45f412f450 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.748+0000 7f45f9a94700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45f409d620 0x7f45f409c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fa295700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f412f450 msgr2=0x7f45f411cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fa295700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f412f450 0x7f45f411cc10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45f406a7d0 tx=0x7f45ec016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec006af0 con 0x7f45f412f450 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: dump: 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.767 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 5 monclient: authenticate success, global_id 4370 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 10 monclient: _renew_subs 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec020070 con 0x7f45f412f450 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f411f270 con 0x7f45f412f450 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec01b450 con 0x7f45f412f450 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f37fe700 4 set_mon_vals no callback set 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 10 monclient: dump: 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.769 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 10 monclient: _renew_subs 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f411f5a0 con 0x7f45f412f450 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f45ec022030 con 0x7f45f412f450 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> v2:172.21.15.174:6812/17603 conn(0x7f45d4035a80 0x7f45d4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 0x7f45f411d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40686b0 con 0x7f45f413f4d0 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f45ec049dd0 con 0x7f45f412f450 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 0x7f45f411d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9a94700 1 --2- 172.21.15.104:0/2483664379 >> v2:172.21.15.174:6812/17603 conn(0x7f45d4035a80 0x7f45d4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient: get_auth_request con 0x7f45f413f4d0 auth_method 0 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9a94700 10 monclient: get_auth_request con 0x7f45d4035a80 auth_method 0 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-09-18T14:21:41.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.752+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45f4064b30 con 0x7f45f413f4d0 2024-09-18T14:21:41.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f9a94700 1 --2- 172.21.15.104:0/2483664379 >> v2:172.21.15.174:6812/17603 conn(0x7f45d4035a80 0x7f45d4037f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009fd0 tx=0x7f45e40091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 msgr2=0x7f45f411d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 0x7f45f411d150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f45dc007970 tx=0x7f45dc0053c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc007b20 con 0x7f45f413f4d0 2024-09-18T14:21:41.772 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 10 monclient: dump: 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f45dc006020 con 0x7f45f413f4d0 2024-09-18T14:21:41.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:41.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 msgr2=0x7f45f411d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 0x7f45f411d150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f45dc007970 tx=0x7f45dc0053c0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 msgr2=0x7f45f411d150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.756+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f413f4d0 0x7f45f411d150 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45fbcf8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:21:41.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 0x7f45f411d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45fbcf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f411f400 con 0x7f45f412e5f0 2024-09-18T14:21:41.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 0x7f45f411d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient: get_auth_request con 0x7f45f412e5f0 auth_method 0 2024-09-18T14:21:41.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-09-18T14:21:41.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f45f4096bf0 con 0x7f45f412e5f0 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 msgr2=0x7f45f411d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 0x7f45f411d690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f45dc017000 tx=0x7f45dc018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 msgr2=0x7f45f411d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 msgr2=0x7f45f411d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 0x7f45f411d690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f45dc017000 tx=0x7f45dc018000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 0x7f45f411d690 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f45dc017000 tx=0x7f45dc018000 comp rx=0 tx=0).stop 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 msgr2=0x7f45f411d690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.772+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 0x7f45f411d690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 0x7f45d403e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 msgr2=0x7f45f411d690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f412e5f0 0x7f45f411d690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d403e770 con 0x7f45d403bd80 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 0x7f45d403e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient: get_auth_request con 0x7f45d403bd80 auth_method 0 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2024-09-18T14:21:41.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f45dc006020 con 0x7f45d403bd80 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 msgr2=0x7f45d403e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f9293700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 0x7f45d403e160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f45f4098330 tx=0x7f45dc023040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc031070 con 0x7f45d403bd80 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.776+0000 7f45f1ffb700 10 monclient: dump: 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.789 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f45dc01c980 con 0x7f45d403bd80 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45f1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45f1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 msgr2=0x7f45d403e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 0x7f45d403e160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f45f4098330 tx=0x7f45dc023040 comp rx=0 tx=0).stop 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45f1ffb700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 msgr2=0x7f45d403e160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fa295700 1 -- 172.21.15.104:0/2483664379 reap_dead start 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45f1ffb700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d403bd80 0x7f45d403e160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.790 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 8, 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 22, 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:21:41.791 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.792 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:21:41.793 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:41.794 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:21:41.795 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:21:41.799 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.800 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.801 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 7 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:21:41.802 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:21:41.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 >> v2:172.21.15.174:6812/17603 conn(0x7f45d4035a80 msgr2=0x7f45d4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/2483664379 >> v2:172.21.15.174:6812/17603 conn(0x7f45d4035a80 0x7f45d4037f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009fd0 tx=0x7f45e40091a0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 10 monclient: shutdown 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f412f450 msgr2=0x7f45f411cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f412f450 0x7f45f411cc10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45f406a7d0 tx=0x7f45ec016000 comp rx=0 tx=0).stop 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 shutdown_connections 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/2483664379 >> v2:172.21.15.174:6812/17603 conn(0x7f45d4035a80 0x7f45d4037f30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 --2- 172.21.15.104:0/2483664379 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f45f412f450 0x7f45f411cc10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 >> 172.21.15.104:0/2483664379 conn(0x7f45f40a1290 msgr2=0x7f45f406e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 shutdown_connections 2024-09-18T14:21:41.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.780+0000 7f45fbcf8700 1 -- 172.21.15.104:0/2483664379 wait complete. 2024-09-18T14:21:41.805 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:21:41.806 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T14:21:41.806 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T14:21:41.806 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T14:21:41.806 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T14:21:41.806 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T14:21:41.806 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:21:41.822 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T14:21:41.822 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T14:21:41.822 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:21:41.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient: get_monmap_and_config 2024-09-18T14:21:41.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient: monmap: 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:41.924634+0000 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:41.924634+0000 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:41.928 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 1 Processor -- start 2024-09-18T14:21:41.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 1 -- start start 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient: init 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e101320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient(hunting): picked mon.noname-b con 0x7f9e10131d00 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e1009e830 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9e10136ef0 0x7f9e101392d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient(hunting): picked mon.noname-c con 0x7f9e10136ef0 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1009bdd0 con 0x7f9e1009e830 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e10099f10 con 0x7f9e10136ef0 2024-09-18T14:21:41.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1009b530 con 0x7f9e10131d00 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e16b3f700 10 monclient(hunting): authenticate will time out at 822.914341s 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e101320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e101320d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42424/0 (socket says 172.21.15.104:42424) 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 -- 172.21.15.104:0/2544154271 learned_addr learned my addr 172.21.15.104:0/2544154271 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): get_auth_request con 0x7f9e10131d00 auth_method 0 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1633e700 1 --2- 172.21.15.104:0/2544154271 >> v2:172.21.15.174:3300/0 conn(0x7f9e10136ef0 0x7f9e101392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1633e700 10 monclient(hunting): get_auth_request con 0x7f9e10136ef0 auth_method 0 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1633e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1633e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1633e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1533c700 1 -- 172.21.15.104:0/2544154271 >> v1:172.21.15.174:6789/0 conn(0x7f9e1009e830 legacy=0x7f9e1009ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1533c700 1 -- 172.21.15.104:0/2544154271 >> v1:172.21.15.174:6789/0 conn(0x7f9e1009e830 legacy=0x7f9e1009ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:41.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e1533c700 1 --1- 172.21.15.104:0/2544154271 >> v1:172.21.15.174:6789/0 conn(0x7f9e1009e830 0x7f9e1009ec20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 -- 172.21.15.104:0/2544154271 >> v2:172.21.15.174:3300/0 conn(0x7f9e10136ef0 msgr2=0x7f9e101392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 --2- 172.21.15.104:0/2544154271 >> v2:172.21.15.174:3300/0 conn(0x7f9e10136ef0 0x7f9e101392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 -- 172.21.15.104:0/2544154271 >> v1:172.21.15.174:6789/0 conn(0x7f9e1009e830 legacy=0x7f9e1009ec20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 monclient: found mon.noname-b 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.920+0000 7f9e15b3d700 1 -- 172.21.15.104:0/2544154271 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1009b810 con 0x7f9e10131d00 2024-09-18T14:21:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e15b3d700 1 -- 172.21.15.104:0/2544154271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 msgr2=0x7f9e101320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e15b3d700 1 --2- 172.21.15.104:0/2544154271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e101320d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009f20 tx=0x7f9e0c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13a5466540cc9fa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 1 -- 172.21.15.104:0/2544154271 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c00e070 con 0x7f9e10131d00 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: dump: 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.943 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 5 monclient: authenticate success, global_id 4373 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 1 -- 172.21.15.104:0/2544154271 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c00bc90 con 0x7f9e10131d00 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 4 set_mon_vals no callback set 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 1 -- 172.21.15.104:0/2544154271 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c0092e0 con 0x7f9e10131d00 2024-09-18T14:21:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e14b3b700 10 monclient: dump: 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 10 monclient: shutdown 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 -- 172.21.15.104:0/2544154271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 msgr2=0x7f9e101320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/2544154271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e101320d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009f20 tx=0x7f9e0c009840 comp rx=0 tx=0).stop 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 -- 172.21.15.104:0/2544154271 shutdown_connections 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/2544154271 >> v2:172.21.15.174:3300/0 conn(0x7f9e10136ef0 0x7f9e101392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/2544154271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e101320d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 -- 172.21.15.104:0/2544154271 >> 172.21.15.104:0/2544154271 conn(0x7f9e100a5330 msgr2=0x7f9e100a5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 -- 172.21.15.104:0/2544154271 shutdown_connections 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 -- 172.21.15.104:0/2544154271 wait complete. 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 10 monclient: build_initial_monmap 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 10 monclient: monmap: 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:41.946 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 Processor -- start 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 1 -- start start 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.924+0000 7f9e16b3f700 10 monclient: init 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e1009e830 0x7f9e10141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): picked mon.noname-a con 0x7f9e1009e830 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e10142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): picked mon.noname-c con 0x7f9e10131d00 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e10136ef0 0x7f9e1006b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): picked mon.noname-b con 0x7f9e10136ef0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1006e300 con 0x7f9e1009e830 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1009f6e0 con 0x7f9e10136ef0 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1009d720 con 0x7f9e10131d00 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e15b3d700 1 -- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e1009e830 msgr2=0x7f9e10141c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e16b3f700 10 monclient(hunting): authenticate will time out at 822.920750s 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e15b3d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e1009e830 0x7f9e10141c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e10142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e10142160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42426/0 (socket says 172.21.15.104:42426) 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 -- 172.21.15.104:0/3453115715 learned_addr learned my addr 172.21.15.104:0/3453115715 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): get_auth_request con 0x7f9e10131d00 auth_method 0 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1633e700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e10136ef0 0x7f9e1006b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1633e700 10 monclient(hunting): get_auth_request con 0x7f9e10136ef0 auth_method 0 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1633e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1633e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1633e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 -- 172.21.15.104:0/3453115715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e10136ef0 msgr2=0x7f9e1006b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e10136ef0 0x7f9e1006b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 -- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e1009e830 msgr2=0x7f9e10141c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e1009e830 0x7f9e10141c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 monclient: found mon.noname-c 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 -- 172.21.15.104:0/3453115715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c009ae0 con 0x7f9e10131d00 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 -- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 msgr2=0x7f9e10142160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e1533c700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e10142160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000c5e0 tx=0x7f9e00007f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.928+0000 7f9e05ffb700 1 -- 172.21.15.104:0/3453115715 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e000074f0 con 0x7f9e10131d00 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: dump: 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: _finish_auth 0 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 5 monclient: authenticate success, global_id 4376 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 1 -- 172.21.15.104:0/3453115715 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e00007650 con 0x7f9e10131d00 2024-09-18T14:21:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 10 monclient: _renew_subs 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1006d9c0 con 0x7f9e10131d00 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 1 -- 172.21.15.104:0/3453115715 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00005a60 con 0x7f9e10131d00 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 10 monclient: dump: 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:41.953 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e07fff700 4 set_mon_vals no callback set 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 10 monclient: _renew_subs 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1006d4b0 con 0x7f9e10131d00 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e100633b0 con 0x7f9e10131d00 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 1 -- 172.21.15.104:0/3453115715 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f9e0001b3b0 con 0x7f9e10131d00 2024-09-18T14:21:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 1 --2- 172.21.15.104:0/3453115715 >> v2:172.21.15.174:6812/17603 conn(0x7f9df4035a30 0x7f9df4037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e05ffb700 1 -- 172.21.15.104:0/3453115715 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f9e00047690 con 0x7f9e10131d00 2024-09-18T14:21:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e15b3d700 1 --2- 172.21.15.104:0/3453115715 >> v2:172.21.15.174:6812/17603 conn(0x7f9df4035a30 0x7f9df4037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.932+0000 7f9e15b3d700 10 monclient: get_auth_request con 0x7f9df4035a30 auth_method 0 2024-09-18T14:21:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.936+0000 7f9e15b3d700 1 --2- 172.21.15.104:0/3453115715 >> v2:172.21.15.174:6812/17603 conn(0x7f9df4035a30 0x7f9df4037ee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009f20 tx=0x7f9e0c017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.940+0000 7f9e05ffb700 1 -- 172.21.15.104:0/3453115715 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9e0000f650 con 0x7f9e10131d00 2024-09-18T14:21:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.940+0000 7f9e05ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:41.940+0000 7f9e05ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:42.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:21:42.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:42.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:42.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e1006ebe0 con 0x7f9e10131d00 2024-09-18T14:21:42.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e05ffb700 1 -- 172.21.15.104:0/3453115715 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f9e00017070 con 0x7f9e10131d00 2024-09-18T14:21:42.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e05ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:42.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e05ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:42.172 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:21:42.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 >> v2:172.21.15.174:6812/17603 conn(0x7f9df4035a30 msgr2=0x7f9df4037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:42.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/3453115715 >> v2:172.21.15.174:6812/17603 conn(0x7f9df4035a30 0x7f9df4037ee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009f20 tx=0x7f9e0c017000 comp rx=0 tx=0).stop 2024-09-18T14:21:42.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 10 monclient: shutdown 2024-09-18T14:21:42.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 msgr2=0x7f9e10142160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e10142160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000c5e0 tx=0x7f9e00007f30 comp rx=0 tx=0).stop 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 shutdown_connections 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/3453115715 >> v2:172.21.15.174:6812/17603 conn(0x7f9df4035a30 0x7f9df4037ee0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e10136ef0 0x7f9e1006b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9e10131d00 0x7f9e10142160 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 --2- 172.21.15.104:0/3453115715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e1009e830 0x7f9e10141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.164+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 >> 172.21.15.104:0/3453115715 conn(0x7f9e100a5330 msgr2=0x7f9e100a3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.168+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 shutdown_connections 2024-09-18T14:21:42.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:42.168+0000 7f9e16b3f700 1 -- 172.21.15.104:0/3453115715 wait complete. 2024-09-18T14:21:45.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.288+0000 7f5946382700 10 monclient: get_monmap_and_config 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.288+0000 7f5946382700 10 monclient: build_initial_monmap 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.288+0000 7f5946382700 10 monclient: monmap: 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:45.294880+0000 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:45.294880+0000 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:45.298 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.288+0000 7f5946382700 1 Processor -- start 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.288+0000 7f5946382700 1 -- start start 2024-09-18T14:21:45.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.288+0000 7f5946382700 10 monclient: init 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59400676a0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f594006f140 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f594006f540 0x7f594013a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient(hunting): picked mon.noname-c con 0x7f594006f540 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594009a1a0 con 0x7f59400676a0 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5940067a70 con 0x7f594006f140 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594009a550 con 0x7f594006f540 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:45.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5946382700 10 monclient(hunting): authenticate will time out at 826.284560s 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945380700 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f59400676a0 legacy=0x7f594009e5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f594006f540 0x7f594013a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5944b7f700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f594006f140 0x7f5940136f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.104:51894/0 (socket says 172.21.15.104:51894) 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5944b7f700 1 -- 172.21.15.104:0/2291091729 learned_addr learned my addr 172.21.15.104:0/2291091729 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): get_auth_request con 0x7f594006f540 auth_method 0 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2024-09-18T14:21:45.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 1 -- 172.21.15.104:0/2291091729 >> v1:172.21.15.174:6789/0 conn(0x7f594006f140 legacy=0x7f5940136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:45.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 1 -- 172.21.15.104:0/2291091729 >> v1:172.21.15.104:6789/0 conn(0x7f59400676a0 legacy=0x7f594009e5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:45.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient: _finish_hunting 0 2024-09-18T14:21:45.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 1 monclient: found mon.noname-c 2024-09-18T14:21:45.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:45.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:45.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.292+0000 7f5945b81700 1 -- 172.21.15.104:0/2291091729 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5940099890 con 0x7f594006f540 2024-09-18T14:21:45.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945b81700 1 -- 172.21.15.104:0/2291091729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f594006f540 msgr2=0x7f594013a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:45.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945b81700 1 --2- 172.21.15.104:0/2291091729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f594006f540 0x7f594013a630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5938009e60 tx=0x7f5938009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e44c0d1668b1c6ab server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:45.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 1 -- 172.21.15.104:0/2291091729 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593800e070 con 0x7f594006f540 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: dump: 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 1 -- 172.21.15.104:0/2291091729 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5938012430 con 0x7f594006f540 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 5 monclient: authenticate success, global_id 4399 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 4 set_mon_vals no callback set 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 1 -- 172.21.15.104:0/2291091729 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59380127d0 con 0x7f594006f540 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5933fff700 10 monclient: dump: 2024-09-18T14:21:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient: shutdown 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- 172.21.15.104:0/2291091729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f594006f540 msgr2=0x7f594013a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 --2- 172.21.15.104:0/2291091729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f594006f540 0x7f594013a630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5938009e60 tx=0x7f5938009780 comp rx=0 tx=0).stop 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- 172.21.15.104:0/2291091729 shutdown_connections 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 --2- 172.21.15.104:0/2291091729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f594006f540 0x7f594013a630 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- 172.21.15.104:0/2291091729 >> 172.21.15.104:0/2291091729 conn(0x7f5940069100 msgr2=0x7f59400a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- 172.21.15.104:0/2291091729 shutdown_connections 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- 172.21.15.104:0/2291091729 wait complete. 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient: build_initial_monmap 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient: monmap: 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 Processor -- start 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- start start 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient: init 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59400676a0 0x7f594011a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): picked mon.noname-b con 0x7f59400676a0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 0x7f594011a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): picked mon.noname-c con 0x7f594006f140 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f594006f540 0x7f594011ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): picked mon.noname-a con 0x7f594006f540 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940126c20 con 0x7f594006f540 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940125590 con 0x7f59400676a0 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5940098a80 con 0x7f594006f140 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5946382700 10 monclient(hunting): authenticate will time out at 826.291024s 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945b81700 1 -- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f594006f540 msgr2=0x7f594011ae00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945b81700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f594006f540 0x7f594011ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 0x7f594011a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 0x7f594011a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42432/0 (socket says 172.21.15.104:42432) 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 -- 172.21.15.104:0/1054299034 learned_addr learned my addr 172.21.15.104:0/1054299034 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): get_auth_request con 0x7f594006f140 auth_method 0 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59400676a0 0x7f594011a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 10 monclient(hunting): get_auth_request con 0x7f59400676a0 auth_method 0 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5945380700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 -- 172.21.15.104:0/1054299034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59400676a0 msgr2=0x7f594011a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59400676a0 0x7f594011a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 -- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f594006f540 msgr2=0x7f594011ae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f594006f540 0x7f594011ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient: _finish_hunting 0 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 monclient: found mon.noname-c 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:45.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.296+0000 7f5944b7f700 1 -- 172.21.15.104:0/1054299034 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5938009a20 con 0x7f594006f140 2024-09-18T14:21:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5944b7f700 1 -- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 msgr2=0x7f594011a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5944b7f700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 0x7f594011a8c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f593400c610 tx=0x7f5934004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 1 -- 172.21.15.104:0/1054299034 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934005d40 con 0x7f594006f140 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: dump: 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:45.318 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: _finish_auth 0 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5946382700 5 monclient: authenticate success, global_id 4379 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 1 -- 172.21.15.104:0/1054299034 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59340070d0 con 0x7f594006f140 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5946382700 10 monclient: _renew_subs 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5946382700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59400a0b50 con 0x7f594006f140 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 1 -- 172.21.15.104:0/1054299034 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934007490 con 0x7f594006f140 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59317fa700 10 monclient: dump: 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f59337fe700 4 set_mon_vals no callback set 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5946382700 10 monclient: _renew_subs 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5946382700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.300+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594011bcd0 con 0x7f594006f140 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f59317fa700 1 -- 172.21.15.104:0/1054299034 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f593401b3b0 con 0x7f594006f140 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f5946382700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f5946382700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f5946382700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59400633b0 con 0x7f594006f140 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f59317fa700 1 --2- 172.21.15.104:0/1054299034 >> v2:172.21.15.174:6812/17603 conn(0x7f5924035a30 0x7f5924037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f59317fa700 1 -- 172.21.15.104:0/1054299034 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f59340474e0 con 0x7f594006f140 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f5945380700 1 --2- 172.21.15.104:0/1054299034 >> v2:172.21.15.174:6812/17603 conn(0x7f5924035a30 0x7f5924037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f5945380700 10 monclient: get_auth_request con 0x7f5924035a30 auth_method 0 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.304+0000 7f5945380700 1 --2- 172.21.15.104:0/1054299034 >> v2:172.21.15.174:6812/17603 conn(0x7f5924035a30 0x7f5924037ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f593c0038e0 tx=0x7f593c009120 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.308+0000 7f59317fa700 1 -- 172.21.15.104:0/1054299034 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f593401a360 con 0x7f594006f140 2024-09-18T14:21:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.308+0000 7f59317fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:45.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.308+0000 7f59317fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:45.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.532+0000 7f5946382700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:21:45.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.532+0000 7f5946382700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:45.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.532+0000 7f5946382700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:45.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.532+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f594006d850 con 0x7f594006f140 2024-09-18T14:21:45.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.536+0000 7f59317fa700 1 -- 172.21.15.104:0/1054299034 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f5934017070 con 0x7f594006f140 2024-09-18T14:21:45.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.536+0000 7f59317fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:45.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.536+0000 7f59317fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:45.544 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:21:45.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.536+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 >> v2:172.21.15.174:6812/17603 conn(0x7f5924035a30 msgr2=0x7f5924037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:45.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 --2- 172.21.15.104:0/1054299034 >> v2:172.21.15.174:6812/17603 conn(0x7f5924035a30 0x7f5924037ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f593c0038e0 tx=0x7f593c009120 comp rx=0 tx=0).stop 2024-09-18T14:21:45.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 10 monclient: shutdown 2024-09-18T14:21:45.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:45.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 msgr2=0x7f594011a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 0x7f594011a8c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f593400c610 tx=0x7f5934004a80 comp rx=0 tx=0).stop 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 shutdown_connections 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 --2- 172.21.15.104:0/1054299034 >> v2:172.21.15.174:6812/17603 conn(0x7f5924035a30 0x7f5924037ee0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f594006f540 0x7f594011ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f594006f140 0x7f594011a8c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 --2- 172.21.15.104:0/1054299034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59400676a0 0x7f594011a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 >> 172.21.15.104:0/1054299034 conn(0x7f5940069100 msgr2=0x7f5940097a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:45.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 shutdown_connections 2024-09-18T14:21:45.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:45.540+0000 7f5946382700 1 -- 172.21.15.104:0/1054299034 wait complete. 2024-09-18T14:21:48.564 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:21:48.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.668+0000 7f2bf759e700 10 monclient: get_monmap_and_config 2024-09-18T14:21:48.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.668+0000 7f2bf759e700 10 monclient: build_initial_monmap 2024-09-18T14:21:48.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.668+0000 7f2bf759e700 10 monclient: monmap: 2024-09-18T14:21:48.677 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:48.674353+0000 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:48.674353+0000 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:48.678 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:48.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.668+0000 7f2bf759e700 1 Processor -- start 2024-09-18T14:21:48.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.668+0000 7f2bf759e700 1 -- start start 2024-09-18T14:21:48.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.668+0000 7f2bf759e700 10 monclient: init 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf809ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient(hunting): picked mon.noname-b con 0x7f2bf80a59d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2bf809bea0 0x7f2bf809f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient(hunting): picked mon.noname-a con 0x7f2bf809bea0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bf809fcb0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf809f780 con 0x7f2bf809fcb0 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf809af10 con 0x7f2bf809bea0 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:48.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf80a1870 con 0x7f2bf80a59d0 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient(hunting): authenticate will time out at 829.664495s 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf5d9b700 1 -- >> v2:172.21.15.104:3300/0 conn(0x7f2bf809bea0 msgr2=0x7f2bf809f240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf5d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2bf809bea0 0x7f2bf809f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf809ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf809ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42434/0 (socket says 172.21.15.104:42434) 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 -- 172.21.15.104:0/2510950821 learned_addr learned my addr 172.21.15.104:0/2510950821 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): get_auth_request con 0x7f2bf80a59d0 auth_method 0 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:48.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient(hunting): handle_auth_done global_id 4382 payload 274 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 -- 172.21.15.104:0/2510950821 >> v2:172.21.15.104:3300/0 conn(0x7f2bf809bea0 msgr2=0x7f2bf809f240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 --2- 172.21.15.104:0/2510950821 >> v2:172.21.15.104:3300/0 conn(0x7f2bf809bea0 0x7f2bf809f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 -- 172.21.15.104:0/2510950821 >> v1:172.21.15.174:6789/0 conn(0x7f2bf809fcb0 legacy=0x7f2bf80a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient: _finish_hunting 0 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 monclient: found mon.noname-b 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:48.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 -- 172.21.15.104:0/2510950821 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80a5da0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 -- 172.21.15.104:0/2510950821 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 msgr2=0x7f2bf809ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf659c700 1 --2- 172.21.15.104:0/2510950821 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf809ed00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2be80099b0 tx=0x7f2be8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e821c6996267b51e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 1 -- 172.21.15.104:0/2510950821 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be800bb50 con 0x7f2bf80a59d0 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: dump: 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:48.687 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: _finish_auth 0 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 5 monclient: authenticate success, global_id 4382 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 1 -- 172.21.15.104:0/2510950821 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be800bcb0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 1 -- 172.21.15.104:0/2510950821 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8011670 con 0x7f2bf80a59d0 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf759e700 4 set_mon_vals no callback set 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.672+0000 7f2bf559a700 10 monclient: dump: 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:48.689 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient: shutdown 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- 172.21.15.104:0/2510950821 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 msgr2=0x7f2bf809ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 --2- 172.21.15.104:0/2510950821 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf809ed00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2be80099b0 tx=0x7f2be8009250 comp rx=0 tx=0).stop 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- 172.21.15.104:0/2510950821 shutdown_connections 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 --2- 172.21.15.104:0/2510950821 >> v2:172.21.15.104:3300/0 conn(0x7f2bf809bea0 0x7f2bf809f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 --2- 172.21.15.104:0/2510950821 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf809ed00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- 172.21.15.104:0/2510950821 >> 172.21.15.104:0/2510950821 conn(0x7f2bf8067c10 msgr2=0x7f2bf80a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:48.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- 172.21.15.104:0/2510950821 shutdown_connections 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- 172.21.15.104:0/2510950821 wait complete. 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient: build_initial_monmap 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient: monmap: 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:48.691 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 Processor -- start 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- start start 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient: init 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf809bea0 0x7f2bf81411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): picked mon.noname-b con 0x7f2bf809bea0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bf809fcb0 0x7f2bf8131dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): picked mon.noname-a con 0x7f2bf809fcb0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf811b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): picked mon.noname-c con 0x7f2bf80a59d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf811d940 con 0x7f2bf809fcb0 2024-09-18T14:21:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf809dd60 con 0x7f2bf809bea0 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf806f3c0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf759e700 10 monclient(hunting): authenticate will time out at 829.671379s 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf5d9b700 1 -- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bf809fcb0 msgr2=0x7f2bf8131dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf5d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bf809fcb0 0x7f2bf8131dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf811b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf811b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42446/0 (socket says 172.21.15.104:42446) 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 1 -- 172.21.15.104:0/1778870131 learned_addr learned my addr 172.21.15.104:0/1778870131 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 10 monclient(hunting): get_auth_request con 0x7f2bf80a59d0 auth_method 0 2024-09-18T14:21:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf809bea0 0x7f2bf81411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 10 monclient(hunting): get_auth_request con 0x7f2bf809bea0 auth_method 0 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.676+0000 7f2bf659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 -- 172.21.15.104:0/1778870131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf809bea0 msgr2=0x7f2bf81411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf809bea0 0x7f2bf81411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 -- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bf809fcb0 msgr2=0x7f2bf8131dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bf809fcb0 0x7f2bf8131dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 monclient: found mon.noname-c 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 -- 172.21.15.104:0/1778870131 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80094f0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 -- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 msgr2=0x7f2bf811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:48.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf6d9d700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf811b940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000c930 tx=0x7f2bf00026e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:48.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0012840 con 0x7f2bf80a59d0 2024-09-18T14:21:48.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:48.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:48.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: dump: 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: _finish_auth 0 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf00129c0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf759e700 5 monclient: authenticate success, global_id 4385 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf759e700 10 monclient: _renew_subs 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0012d80 con 0x7f2bf80a59d0 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf811ef50 con 0x7f2bf80a59d0 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf4d99700 4 set_mon_vals no callback set 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2be67fc700 10 monclient: dump: 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.699 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf759e700 10 monclient: _renew_subs 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.680+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf8141f60 con 0x7f2bf80a59d0 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2bf759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2bf759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2bf759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf811d450 con 0x7f2bf80a59d0 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f2bf000a5c0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2be67fc700 1 --2- 172.21.15.104:0/1778870131 >> v2:172.21.15.174:6812/17603 conn(0x7f2bd8035a30 0x7f2bd8037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f2bf0047d00 con 0x7f2bf80a59d0 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2bf659c700 1 --2- 172.21.15.104:0/1778870131 >> v2:172.21.15.174:6812/17603 conn(0x7f2bd8035a30 0x7f2bd8037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.684+0000 7f2bf659c700 10 monclient: get_auth_request con 0x7f2bd8035a30 auth_method 0 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.688+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 DispatchQueue::entry inject delay of 0.002000 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.688+0000 7f2bf659c700 1 --2- 172.21.15.104:0/1778870131 >> v2:172.21.15.174:6812/17603 conn(0x7f2bd8035a30 0x7f2bd8037ee0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2be80099b0 tx=0x7f2be8016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.692+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2bf0018880 con 0x7f2bf80a59d0 2024-09-18T14:21:48.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.692+0000 7f2be67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:48.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.692+0000 7f2be67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:48.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.904+0000 7f2bf759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:21:48.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.904+0000 7f2bf759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:48.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.904+0000 7f2bf759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:48.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.904+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2bf80633b0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.904+0000 7f2be67fc700 1 -- 172.21.15.104:0/1778870131 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2bf00182c0 con 0x7f2bf80a59d0 2024-09-18T14:21:48.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.904+0000 7f2be67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:48.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.904+0000 7f2be67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:48.915 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:21:48.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 >> v2:172.21.15.174:6812/17603 conn(0x7f2bd8035a30 msgr2=0x7f2bd8037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:48.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 --2- 172.21.15.104:0/1778870131 >> v2:172.21.15.174:6812/17603 conn(0x7f2bd8035a30 0x7f2bd8037ee0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2be80099b0 tx=0x7f2be8016000 comp rx=0 tx=0).stop 2024-09-18T14:21:48.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 10 monclient: shutdown 2024-09-18T14:21:48.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 msgr2=0x7f2bf811b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf811b940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000c930 tx=0x7f2bf00026e0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 shutdown_connections 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 --2- 172.21.15.104:0/1778870131 >> v2:172.21.15.174:6812/17603 conn(0x7f2bd8035a30 0x7f2bd8037ee0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2bf80a59d0 0x7f2bf811b940 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2bf809fcb0 0x7f2bf8131dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 --2- 172.21.15.104:0/1778870131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf809bea0 0x7f2bf81411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 >> 172.21.15.104:0/1778870131 conn(0x7f2bf8067c10 msgr2=0x7f2bf809e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:48.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 shutdown_connections 2024-09-18T14:21:48.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:48.908+0000 7f2bf759e700 1 -- 172.21.15.104:0/1778870131 wait complete. 2024-09-18T14:21:51.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.028+0000 7f315a442700 10 monclient: get_monmap_and_config 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.028+0000 7f315a442700 10 monclient: build_initial_monmap 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.028+0000 7f315a442700 10 monclient: monmap: 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:52.034996+0000 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:52.034996+0000 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:52.038 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:52.039 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:52.039 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:52.039 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:52.039 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 1 Processor -- start 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 1 -- start start 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient: init 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 0x7f315409ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient(hunting): picked mon.noname-a con 0x7f315409a5b0 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f315412f980 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:52.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient(hunting): picked mon.noname-b con 0x7f3154130490 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3154097d10 con 0x7f315412f980 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3154098470 con 0x7f315409a5b0 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3154099650 con 0x7f3154130490 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f315a442700 10 monclient(hunting): authenticate will time out at 833.024720s 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3158c3f700 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f315412f980 legacy=0x7f315412fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159440700 1 -- >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 msgr2=0x7f315409ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159440700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 0x7f315409ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43742/0 (socket says 172.21.15.104:43742) 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 learned_addr learned my addr 172.21.15.104:0/3558398888 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:52.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 10 monclient(hunting): get_auth_request con 0x7f3154130490 auth_method 0 2024-09-18T14:21:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.032+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159440700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 msgr2=0x7f315409ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v1:172.21.15.104:6789/0 conn(0x7f315412f980 legacy=0x7f315412fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-09-18T14:21:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159440700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 0x7f315409ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:21:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 10 monclient(hunting): get_auth_request con 0x7f3154130490 auth_method 0 2024-09-18T14:21:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.236+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:21:52.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159440700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 msgr2=0x7f315409ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:52.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v1:172.21.15.104:6789/0 conn(0x7f315412f980 legacy=0x7f315412fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-09-18T14:21:52.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159440700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 0x7f315409ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:21:52.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 10 monclient(hunting): get_auth_request con 0x7f3154130490 auth_method 0 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:52.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:52.640+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:21:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159440700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 msgr2=0x7f315409ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v1:172.21.15.104:6789/0 conn(0x7f315412f980 legacy=0x7f315412fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-09-18T14:21:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159440700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 0x7f315409ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 10 monclient(hunting): get_auth_request con 0x7f3154130490 auth_method 0 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:53.444+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:21:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 10 monclient(hunting): tick 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 monclient(hunting): continuing hunt 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 msgr2=0x7f3154136ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3154130490 0x7f3154136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 msgr2=0x7f315409ae20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.104:3300/0 conn(0x7f315409a5b0 0x7f315409ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 -- 172.21.15.104:0/3558398888 >> v1:172.21.15.104:6789/0 conn(0x7f315412f980 legacy=0x7f315412fd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 10 monclient: _add_conns ranks=[1,4] 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31380036f0 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 10 monclient(hunting): picked mon.noname-c con 0x7f3138007430 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 -- 172.21.15.104:0/3558398888 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3154099930 con 0x7f31380036f0 2024-09-18T14:21:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:55.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 1 -- 172.21.15.104:0/3558398888 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3154098470 con 0x7f3138007430 2024-09-18T14:21:55.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f314affd700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:55.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:55.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 10 monclient(hunting): get_auth_request con 0x7f3138007430 auth_method 0 2024-09-18T14:21:55.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:55.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:55.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:55.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:55.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:55.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:55.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:55.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.031+0000 7f3159440700 1 -- 172.21.15.104:0/3558398888 >> v1:172.21.15.174:6789/0 conn(0x7f31380036f0 legacy=0x7f3138005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 10 monclient(hunting): get_auth_request con 0x7f3138007430 auth_method 0 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:55.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.235+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:21:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 10 monclient(hunting): get_auth_request con 0x7f3138007430 auth_method 0 2024-09-18T14:21:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:55.639+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:21:56.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:56.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 10 monclient(hunting): get_auth_request con 0x7f3138007430 auth_method 0 2024-09-18T14:21:56.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:56.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:56.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:56.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:21:56.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:56.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:56.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.443+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:21:56.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.583+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255910266 0 0) 0x7f3154099930 con 0x7f31380036f0 2024-09-18T14:21:56.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.583+0000 7f314bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:56.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.583+0000 7f314bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:56.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.583+0000 7f314bfff700 10 monclient(hunting): my global_id is 4408 2024-09-18T14:21:56.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.583+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3140003570 con 0x7f31380036f0 2024-09-18T14:21:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3357329896 0 0) 0x7f3140003570 con 0x7f31380036f0 2024-09-18T14:21:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 msgr2=0x7f31380098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 --2- 172.21.15.104:0/3558398888 >> v2:172.21.15.174:3300/0 conn(0x7f3138007430 0x7f31380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: _finish_hunting 0 2024-09-18T14:21:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 monclient: found mon.noname-c-legacy 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.174:6789/0 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154097d10 con 0x7f31380036f0 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: _finish_auth 0 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7f3144004280 con 0x7f31380036f0 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f315a442700 5 monclient: authenticate success, global_id 4408 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f315a442700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: dump: 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] but i'm connected to v1:172.21.15.174:6789/0, reconnecting 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 >> v1:172.21.15.174:6789/0 conn(0x7f31380036f0 legacy=0x7f3138005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:56.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 1 -- 172.21.15.104:0/3558398888 reap_dead start 2024-09-18T14:21:56.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:21:56.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140003a60 0x7f3140003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): picked mon.a con 0x7f3140003a60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:56.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140004810 0x7f3140004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): picked mon.c con 0x7f3140004810 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:56.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31400052e0 0x7f314000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159440700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140003a60 msgr2=0x7f3140003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:56.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): picked mon.b con 0x7f31400052e0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:56.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3154098470 con 0x7f3140003a60 2024-09-18T14:21:56.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159440700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140003a60 0x7f3140003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:56.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314000c250 con 0x7f31400052e0 2024-09-18T14:21:56.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314000c410 con 0x7f3140004810 2024-09-18T14:21:56.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:56.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f314bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:21:56.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f315a442700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:56.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140004810 0x7f3140004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:56.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): get_auth_request con 0x7f3140004810 auth_method 0 2024-09-18T14:21:56.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:56.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:56.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:56.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31400052e0 0x7f314000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:56.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 10 monclient(hunting): get_auth_request con 0x7f31400052e0 auth_method 0 2024-09-18T14:21:56.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:56.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:56.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:56.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3159c41700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient(hunting): handle_auth_done global_id 4408 payload 293 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31400052e0 msgr2=0x7f314000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:56.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31400052e0 0x7f314000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140003a60 msgr2=0x7f3140003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:56.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140003a60 0x7f3140003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient: _finish_hunting 0 2024-09-18T14:21:56.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 1 monclient: found mon.c 2024-09-18T14:21:56.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:56.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:56.607 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3140003570 con 0x7f3140004810 2024-09-18T14:21:56.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 10 monclient: _finish_auth 0 2024-09-18T14:21:56.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.587+0000 7f3158c3f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:56.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f3158c3f700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140004810 msgr2=0x7f3140004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:56.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f3158c3f700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140004810 0x7f3140004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3150007fb0 tx=0x7f31500077a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50339ca7981657c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:56.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150007ad0 con 0x7f3140004810 2024-09-18T14:21:56.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:56.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:56.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 10 monclient: dump: 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3150011610 con 0x7f3140004810 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f315a442700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f315a442700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 1 -- 172.21.15.104:0/3558398888 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31500119b0 con 0x7f3140004810 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f315a442700 4 set_mon_vals no callback set 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.591+0000 7f314bfff700 10 monclient: dump: 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient: shutdown 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 -- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140004810 msgr2=0x7f3140004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140004810 0x7f3140004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3150007fb0 tx=0x7f31500077a0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 -- 172.21.15.104:0/3558398888 shutdown_connections 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31400052e0 0x7f314000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140004810 0x7f3140004c80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 --2- 172.21.15.104:0/3558398888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140003a60 0x7f3140003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 -- 172.21.15.104:0/3558398888 >> 172.21.15.104:0/3558398888 conn(0x7f31540a12c0 msgr2=0x7f31540a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 -- 172.21.15.104:0/3558398888 shutdown_connections 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 -- 172.21.15.104:0/3558398888 wait complete. 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient: build_initial_monmap 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient: monmap: 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:56.618 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 Processor -- start 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 -- start start 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient: init 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 0x7f3154140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient(hunting): picked mon.noname-c con 0x7f3140003a60 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3140004810 0x7f31541410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient(hunting): picked mon.noname-b con 0x7f3140004810 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31400052e0 0x7f3154095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient(hunting): picked mon.noname-a con 0x7f31400052e0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315409cd50 con 0x7f31400052e0 2024-09-18T14:21:56.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.595+0000 7f315a442700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f315a442700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315411eeb0 con 0x7f3140004810 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f315a442700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f315a442700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315411ebd0 con 0x7f3140003a60 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f315a442700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f315a442700 10 monclient(hunting): authenticate will time out at 837.592049s 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159c41700 1 -- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31400052e0 msgr2=0x7f3154095a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159c41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31400052e0 0x7f3154095a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 0x7f3154140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:56.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 0x7f3154140b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43770/0 (socket says 172.21.15.104:43770) 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 -- 172.21.15.104:0/2573596135 learned_addr learned my addr 172.21.15.104:0/2573596135 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): get_auth_request con 0x7f3140003a60 auth_method 0 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3140004810 0x7f31541410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 10 monclient(hunting): get_auth_request con 0x7f3140004810 auth_method 0 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:56.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3158c3f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 -- 172.21.15.104:0/2573596135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3140004810 msgr2=0x7f31541410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3140004810 0x7f31541410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 -- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31400052e0 msgr2=0x7f3154095a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31400052e0 0x7f3154095a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient: _finish_hunting 0 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 monclient: found mon.noname-c 2024-09-18T14:21:56.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 -- 172.21.15.104:0/2573596135 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315411dc70 con 0x7f3140003a60 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 -- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 msgr2=0x7f3154140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f3159440700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 0x7f3154140b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3144003b50 tx=0x7f31440043f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f31497fa700 1 -- 172.21.15.104:0/2573596135 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144007570 con 0x7f3140003a60 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f31497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f31497fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f31497fa700 10 monclient: dump: 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:56.623 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f31497fa700 10 monclient: _finish_auth 0 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f31497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.599+0000 7f31497fa700 1 -- 172.21.15.104:0/2573596135 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31440047a0 con 0x7f3140003a60 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 5 monclient: authenticate success, global_id 4388 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 10 monclient: _renew_subs 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:56.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315411fea0 con 0x7f3140003a60 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 1 -- 172.21.15.104:0/2573596135 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144004b60 con 0x7f3140003a60 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 10 monclient: dump: 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:56.625 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f314b7fe700 4 set_mon_vals no callback set 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 10 monclient: _renew_subs 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31541417b0 con 0x7f3140003a60 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 1 -- 172.21.15.104:0/2573596135 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f3144014420 con 0x7f3140003a60 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315411de60 con 0x7f3140003a60 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 1 --2- 172.21.15.104:0/2573596135 >> v2:172.21.15.174:6812/17603 conn(0x7f3138036270 0x7f3138038720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f31497fa700 1 -- 172.21.15.104:0/2573596135 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f3144047f70 con 0x7f3140003a60 2024-09-18T14:21:56.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f3158c3f700 1 --2- 172.21.15.104:0/2573596135 >> v2:172.21.15.174:6812/17603 conn(0x7f3138036270 0x7f3138038720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:56.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.603+0000 7f3158c3f700 10 monclient: get_auth_request con 0x7f3138036270 auth_method 0 2024-09-18T14:21:56.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.607+0000 7f3158c3f700 1 --2- 172.21.15.104:0/2573596135 >> v2:172.21.15.174:6812/17603 conn(0x7f3138036270 0x7f3138038720 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f31500077e0 tx=0x7f3150018000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:56.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.615+0000 7f31497fa700 1 -- 172.21.15.104:0/2573596135 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f314401b680 con 0x7f3140003a60 2024-09-18T14:21:56.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.615+0000 7f31497fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:56.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.615+0000 7f31497fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18275]: starting. 2024-09-18T14:21:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18275]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18274 2024-09-18T14:21:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18275]: created object 0... 2024-09-18T14:21:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18275]: created object 25... 2024-09-18T14:21:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18275]: created object 49... 2024-09-18T14:21:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18275]: finishing. 2024-09-18T14:21:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18275]: shutting down. 2024-09-18T14:21:56.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:21:56.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:56.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:56.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f315411ed10 con 0x7f3140003a60 2024-09-18T14:21:56.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f31497fa700 1 -- 172.21.15.104:0/2573596135 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f315411ed10 con 0x7f3140003a60 2024-09-18T14:21:56.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f31497fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:21:56.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f31497fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:56.872 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 >> v2:172.21.15.174:6812/17603 conn(0x7f3138036270 msgr2=0x7f3138038720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 --2- 172.21.15.104:0/2573596135 >> v2:172.21.15.174:6812/17603 conn(0x7f3138036270 0x7f3138038720 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f31500077e0 tx=0x7f3150018000 comp rx=0 tx=0).stop 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 10 monclient: shutdown 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 msgr2=0x7f3154140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 0x7f3154140b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3144003b50 tx=0x7f31440043f0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 shutdown_connections 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 --2- 172.21.15.104:0/2573596135 >> v2:172.21.15.174:6812/17603 conn(0x7f3138036270 0x7f3138038720 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31400052e0 0x7f3154095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3140004810 0x7f31541410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 --2- 172.21.15.104:0/2573596135 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3140003a60 0x7f3154140b90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 >> 172.21.15.104:0/2573596135 conn(0x7f31540a12c0 msgr2=0x7f315409ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:56.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 shutdown_connections 2024-09-18T14:21:56.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.863+0000 7f315a442700 1 -- 172.21.15.104:0/2573596135 wait complete. 2024-09-18T14:21:56.885 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T14:21:56.885 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T14:21:56.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T14:21:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient: get_monmap_and_config 2024-09-18T14:21:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient: build_initial_monmap 2024-09-18T14:21:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient: monmap: 2024-09-18T14:21:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:56.981885+0000 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:56.981885+0000 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:56.986 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 1 Processor -- start 2024-09-18T14:21:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 1 -- start start 2024-09-18T14:21:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient: init 2024-09-18T14:21:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T14:21:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe700a5d10 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbe7009e090 0x7fbe7009cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient(hunting): picked mon.noname-a con 0x7fbe7009e090 addr v2:172.21.15.104:3300/0 2024-09-18T14:21:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient(hunting): picked mon.noname-b con 0x7fbe700a0700 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7009d6b0 con 0x7fbe700a5d10 2024-09-18T14:21:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe700a1ed0 con 0x7fbe7009e090 2024-09-18T14:21:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe700a60e0 con 0x7fbe700a0700 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe778cd700 10 monclient(hunting): authenticate will time out at 837.971845s 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe74e68700 1 -- >> v2:172.21.15.104:3300/0 conn(0x7fbe7009e090 msgr2=0x7fbe7009cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.975+0000 7fbe74e68700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbe7009e090 0x7fbe7009cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43790/0 (socket says 172.21.15.104:43790) 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3441082992 learned_addr learned my addr 172.21.15.104:0/3441082992 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): get_auth_request con 0x7fbe700a0700 auth_method 0 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient(hunting): handle_auth_done global_id 4391 payload 274 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3441082992 >> v2:172.21.15.104:3300/0 conn(0x7fbe7009e090 msgr2=0x7fbe7009cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3441082992 >> v2:172.21.15.104:3300/0 conn(0x7fbe7009e090 0x7fbe7009cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3441082992 >> v1:172.21.15.174:6789/0 conn(0x7fbe700a5d10 legacy=0x7fbe7009fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient: _finish_hunting 0 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 monclient: found mon.noname-b 2024-09-18T14:21:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3441082992 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7009d3d0 con 0x7fbe700a0700 2024-09-18T14:21:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 msgr2=0x7fbe70139790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:21:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 msgr2=0x7fbe70139790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:21:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fbe6c0099b0 tx=0x7fbe6c009250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:56.979+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75e6a700 10 monclient: get_auth_request con 0x7fbe700a0700 auth_method 0 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75e6a700 0 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: ms_handle_reset current mon [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 -- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 msgr2=0x7fbe70139790 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe700a0700 0x7fbe70139790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: _add_conns ranks=[4,0] 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 --2- 172.21.15.104:0/3441082992 >> v2:172.21.15.174:3300/0 conn(0x7fbe5c003550 0x7fbe5c0059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c003550 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:57.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe5c0062f0 addr v1:172.21.15.104:6789/0 2024-09-18T14:21:57.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 -- 172.21.15.104:0/3441082992 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c00a250 con 0x7fbe5c0062f0 2024-09-18T14:21:57.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 -- 172.21.15.104:0/3441082992 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5c00b410 con 0x7fbe5c003550 2024-09-18T14:21:57.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:57.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe74e68700 1 -- 172.21.15.104:0/3441082992 >> v1:172.21.15.104:6789/0 conn(0x7fbe5c0062f0 legacy=0x7fbe5c0087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 1 --2- 172.21.15.104:0/3441082992 >> v2:172.21.15.174:3300/0 conn(0x7fbe5c003550 0x7fbe5c0059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): get_auth_request con 0x7fbe5c003550 auth_method 0 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient(hunting): handle_auth_done global_id 4391 payload 293 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 1 -- 172.21.15.104:0/3441082992 >> v1:172.21.15.104:6789/0 conn(0x7fbe5c0062f0 legacy=0x7fbe5c0087a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:21:57.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient: _finish_hunting 0 2024-09-18T14:21:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 1 monclient: found mon.noname-c 2024-09-18T14:21:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:21:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 1 -- 172.21.15.104:0/3441082992 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c00b640 con 0x7fbe5c003550 2024-09-18T14:21:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 1 -- 172.21.15.104:0/3441082992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe5c003550 msgr2=0x7fbe5c0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75669700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe5c003550 0x7fbe5c0059c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbe68002e50 tx=0x7fbe68002530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c767c8bcbfbbd904 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 -- 172.21.15.104:0/3441082992 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68016070 con 0x7fbe5c003550 2024-09-18T14:21:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: dump: 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 -- 172.21.15.104:0/3441082992 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe68006ba0 con 0x7fbe5c003550 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 5 monclient: authenticate success, global_id 4391 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 1 -- 172.21.15.104:0/3441082992 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68002840 con 0x7fbe5c003550 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 4 set_mon_vals no callback set 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe67fff700 10 monclient: dump: 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.196 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 10 monclient: shutdown 2024-09-18T14:21:57.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 1 -- 172.21.15.104:0/3441082992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe5c003550 msgr2=0x7fbe5c0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe5c003550 0x7fbe5c0059c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbe68002e50 tx=0x7fbe68002530 comp rx=0 tx=0).stop 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3441082992 reap_dead start 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 1 -- 172.21.15.104:0/3441082992 shutdown_connections 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 1 --2- 172.21.15.104:0/3441082992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe5c003550 0x7fbe5c0059c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 1 -- 172.21.15.104:0/3441082992 >> 172.21.15.104:0/3441082992 conn(0x7fbe700695f0 msgr2=0x7fbe700a5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 1 -- 172.21.15.104:0/3441082992 shutdown_connections 2024-09-18T14:21:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.183+0000 7fbe778cd700 1 -- 172.21.15.104:0/3441082992 wait complete. 2024-09-18T14:21:57.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient: build_initial_monmap 2024-09-18T14:21:57.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient: monmap: 2024-09-18T14:21:57.202 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:57.202 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:57.202 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:57.202 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 Processor -- start 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 -- start start 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient: init 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 0x7fbe70141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c003550 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:57.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe70141d80 0x7fbe70142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): picked mon.noname-a con 0x7fbe70141d80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe7008ce80 0x7fbe7008d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): picked mon.noname-b con 0x7fbe7008ce80 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70098580 con 0x7fbe70141d80 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7008eeb0 con 0x7fbe7008ce80 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7008e930 con 0x7fbe5c003550 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe778cd700 10 monclient(hunting): authenticate will time out at 838.183130s 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe74e68700 1 -- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe70141d80 msgr2=0x7fbe70142150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:57.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe74e68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe70141d80 0x7fbe70142150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 0x7fbe70141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 0x7fbe70141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43800/0 (socket says 172.21.15.104:43800) 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 1 -- 172.21.15.104:0/3919514931 learned_addr learned my addr 172.21.15.104:0/3919514931 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 10 monclient(hunting): get_auth_request con 0x7fbe5c003550 auth_method 0 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe7008ce80 0x7fbe7008d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75669700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 10 monclient(hunting): get_auth_request con 0x7fbe7008ce80 auth_method 0 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.187+0000 7fbe75e6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe7008ce80 msgr2=0x7fbe7008d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe7008ce80 0x7fbe7008d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe70141d80 msgr2=0x7fbe70142150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe70141d80 0x7fbe70142150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 10 monclient: _finish_hunting 0 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 monclient: found mon.noname-c 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:57.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 -- 172.21.15.104:0/3919514931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700a4720 con 0x7fbe5c003550 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 msgr2=0x7fbe70141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe75669700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 0x7fbe70141840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00b550 tx=0x7fbe68016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6800c3f0 con 0x7fbe5c003550 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: dump: 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.207 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: _finish_auth 0 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe778cd700 5 monclient: authenticate success, global_id 4394 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe68007670 con 0x7fbe5c003550 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe778cd700 10 monclient: _renew_subs 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68007a10 con 0x7fbe5c003550 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe778cd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70142690 con 0x7fbe5c003550 2024-09-18T14:21:57.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe677fe700 4 set_mon_vals no callback set 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe657fa700 10 monclient: dump: 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe778cd700 10 monclient: _renew_subs 2024-09-18T14:21:57.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe778cd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:57.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.191+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe70090460 con 0x7fbe5c003550 2024-09-18T14:21:57.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe778cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe778cd700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 0x7fbe7008dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe778cd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe700686b0 con 0x7fbe7008d830 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 0x7fbe7008dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient: get_auth_request con 0x7fbe7008d830 auth_method 0 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fbe68024070 con 0x7fbe5c003550 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe74e68700 1 -- 172.21.15.104:0/3919514931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe70090650 con 0x7fbe7008d830 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe657fa700 1 --2- 172.21.15.104:0/3919514931 >> v2:172.21.15.174:6812/17603 conn(0x7fbe54035a30 0x7fbe54037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fbe6804a150 con 0x7fbe5c003550 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3919514931 >> v2:172.21.15.174:6812/17603 conn(0x7fbe54035a30 0x7fbe54037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.195+0000 7fbe75e6a700 10 monclient: get_auth_request con 0x7fbe54035a30 auth_method 0 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe74e68700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 msgr2=0x7fbe7008dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe74e68700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 0x7fbe7008dc00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe600039d0 tx=0x7fbe60009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60006150 con 0x7fbe7008d830 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 10 monclient: dump: 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.213 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbe600062b0 con 0x7fbe7008d830 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 msgr2=0x7fbe7008dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 0x7fbe7008dc00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe600039d0 tx=0x7fbe60009220 comp rx=0 tx=0).stop 2024-09-18T14:21:57.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 msgr2=0x7fbe7008dc00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:57.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe657fa700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7008d830 0x7fbe7008dc00 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe600039d0 tx=0x7fbe60009220 comp rx=0 tx=0).stop 2024-09-18T14:21:57.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.199+0000 7fbe75e6a700 1 --2- 172.21.15.104:0/3919514931 >> v2:172.21.15.174:6812/17603 conn(0x7fbe54035a30 0x7fbe54037ee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbe7006b4d0 tx=0x7fbe6c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe778cd700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe778cd700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe778cd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70064b30 con 0x7fbe7006d910 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe74e68700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe74e68700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 msgr2=0x7fbe7006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe74e68700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 msgr2=0x7fbe7006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe74e68700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:21:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.211+0000 7fbe74e68700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:57.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient: get_auth_request con 0x7fbe7006d910 auth_method 0 2024-09-18T14:21:57.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-09-18T14:21:57.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 1 -- 172.21.15.104:0/3919514931 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbe7006a190 con 0x7fbe7006d910 2024-09-18T14:21:57.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 msgr2=0x7fbe7006dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.415+0000 7fbe74e68700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbe600039d0 tx=0x7fbe60014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe600068f0 con 0x7fbe7006d910 2024-09-18T14:21:57.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 10 monclient: dump: 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.427 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.428 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fbe60020810 con 0x7fbe7006d910 2024-09-18T14:21:57.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:21:57.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:57.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 msgr2=0x7fbe7006dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbe600039d0 tx=0x7fbe60014000 comp rx=0 tx=0).stop 2024-09-18T14:21:57.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 msgr2=0x7fbe7006dd10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:57.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe657fa700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe7006d910 0x7fbe7006dd10 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbe600039d0 tx=0x7fbe60014000 comp rx=0 tx=0).stop 2024-09-18T14:21:57.429 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:21:57.429 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:21:57.429 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:21:57.429 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 12, 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 0, 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:57.431 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:21:57.432 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:21:57.433 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:57.434 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.435 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:21:57.436 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.437 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.438 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.439 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:21:57.440 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:57.441 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 4 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.442 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:21:57.443 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:21:57.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 >> v2:172.21.15.174:6812/17603 conn(0x7fbe54035a30 msgr2=0x7fbe54037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 --2- 172.21.15.104:0/3919514931 >> v2:172.21.15.174:6812/17603 conn(0x7fbe54035a30 0x7fbe54037ee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbe7006b4d0 tx=0x7fbe6c008040 comp rx=0 tx=0).stop 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 10 monclient: shutdown 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 msgr2=0x7fbe70141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 0x7fbe70141840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00b550 tx=0x7fbe68016000 comp rx=0 tx=0).stop 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe75e6a700 1 -- 172.21.15.104:0/3919514931 reap_dead start 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 shutdown_connections 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 --2- 172.21.15.104:0/3919514931 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe5c003550 0x7fbe70141840 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 >> 172.21.15.104:0/3919514931 conn(0x7fbe700695f0 msgr2=0x7fbe700a3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 shutdown_connections 2024-09-18T14:21:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.419+0000 7fbe778cd700 1 -- 172.21.15.104:0/3919514931 wait complete. 2024-09-18T14:21:57.446 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T14:21:57.446 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T14:21:57.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:21:57.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient: get_monmap_and_config 2024-09-18T14:21:57.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient: build_initial_monmap 2024-09-18T14:21:57.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient: monmap: 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:57.557231+0000 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:57.557231+0000 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:21:57.561 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 1 Processor -- start 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 1 -- start start 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient: init 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd94c09e830 0x7fd94c131d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient(hunting): picked mon.noname-c con 0x7fd94c09e830 addr v2:172.21.15.174:3300/0 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 0x7fd94c09ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient(hunting): picked mon.noname-b con 0x7fd94c132260 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd94c09f140 addr v1:172.21.15.174:6789/0 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c09b7a0 con 0x7fd94c09f140 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd94c099f10 con 0x7fd94c09e830 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c09b4c0 con 0x7fd94c132260 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.551+0000 7fd95126e700 10 monclient(hunting): authenticate will time out at 838.547928s 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 0x7fd94c09ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 0x7fd94c09ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43834/0 (socket says 172.21.15.104:43834) 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 -- 172.21.15.104:0/1250386741 learned_addr learned my addr 172.21.15.104:0/1250386741 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): get_auth_request con 0x7fd94c132260 auth_method 0 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94bfff700 1 --2- 172.21.15.104:0/1250386741 >> v2:172.21.15.174:3300/0 conn(0x7fd94c09e830 0x7fd94c131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94bfff700 10 monclient(hunting): get_auth_request con 0x7fd94c09e830 auth_method 0 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_done global_id 4403 payload 274 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 -- 172.21.15.104:0/1250386741 >> v2:172.21.15.174:3300/0 conn(0x7fd94c09e830 msgr2=0x7fd94c131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 --2- 172.21.15.104:0/1250386741 >> v2:172.21.15.174:3300/0 conn(0x7fd94c09e830 0x7fd94c131d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 -- 172.21.15.104:0/1250386741 >> v1:172.21.15.174:6789/0 conn(0x7fd94c09f140 legacy=0x7fd94c09fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 monclient: found mon.noname-b 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:57.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:21:57.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 -- 172.21.15.104:0/1250386741 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c09ba80 con 0x7fd94c132260 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 -- 172.21.15.104:0/1250386741 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 msgr2=0x7fd94c09ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94b7fe700 1 --2- 172.21.15.104:0/1250386741 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 0x7fd94c09ec00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd94c09c4d0 tx=0x7fd93c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7136756117d04d69 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 1 -- 172.21.15.104:0/1250386741 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00ba70 con 0x7fd94c132260 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: dump: 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.569 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: _finish_auth 0 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd95126e700 5 monclient: authenticate success, global_id 4403 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 1 -- 172.21.15.104:0/1250386741 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c00bbd0 con 0x7fd94c132260 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd95126e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd95126e700 10 monclient: get_monmap_and_config success 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd95126e700 4 set_mon_vals no callback set 2024-09-18T14:21:57.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 1 -- 172.21.15.104:0/1250386741 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c011420 con 0x7fd94c132260 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.555+0000 7fd94affd700 10 monclient: dump: 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.571 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient: shutdown 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- 172.21.15.104:0/1250386741 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 msgr2=0x7fd94c09ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 --2- 172.21.15.104:0/1250386741 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 0x7fd94c09ec00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd94c09c4d0 tx=0x7fd93c009250 comp rx=0 tx=0).stop 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- 172.21.15.104:0/1250386741 shutdown_connections 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 --2- 172.21.15.104:0/1250386741 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c132260 0x7fd94c09ec00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 --2- 172.21.15.104:0/1250386741 >> v2:172.21.15.174:3300/0 conn(0x7fd94c09e830 0x7fd94c131d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- 172.21.15.104:0/1250386741 >> 172.21.15.104:0/1250386741 conn(0x7fd94c0676a0 msgr2=0x7fd94c067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- 172.21.15.104:0/1250386741 shutdown_connections 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- 172.21.15.104:0/1250386741 wait complete. 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient: build_initial_monmap 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient: monmap: 2024-09-18T14:21:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 Processor -- start 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- start start 2024-09-18T14:21:57.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient: init 2024-09-18T14:21:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient: _reopen_session rank -1 2024-09-18T14:21:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:21:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c09e830 0x7fd94c130d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): picked mon.noname-b con 0x7fd94c09e830 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:21:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 0x7fd94c141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): picked mon.noname-c con 0x7fd94c09f140 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd94c132260 0x7fd94c141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): picked mon.noname-a con 0x7fd94c132260 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c11cd90 con 0x7fd94c132260 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c09f5e0 con 0x7fd94c09e830 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c11f4c0 con 0x7fd94c09f140 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): _renew_subs 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd95126e700 10 monclient(hunting): authenticate will time out at 838.555206s 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd950a6d700 1 -- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd94c132260 msgr2=0x7fd94c141640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3300/0 2024-09-18T14:21:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd950a6d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd94c132260 0x7fd94c141640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 0x7fd94c141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 0x7fd94c141100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43842/0 (socket says 172.21.15.104:43842) 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c09e830 0x7fd94c130d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 -- 172.21.15.104:0/798226483 learned_addr learned my addr 172.21.15.104:0/798226483 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): get_auth_request con 0x7fd94c09f140 auth_method 0 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 10 monclient(hunting): get_auth_request con 0x7fd94c09e830 auth_method 0 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c09e830 msgr2=0x7fd94c130d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c09e830 0x7fd94c130d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd94c132260 msgr2=0x7fd94c141640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd94c132260 0x7fd94c141640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 monclient: found mon.noname-c 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.559+0000 7fd94b7fe700 1 -- 172.21.15.104:0/798226483 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0094f0 con 0x7fd94c09f140 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd94b7fe700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 msgr2=0x7fd94c141100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd94b7fe700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 0x7fd94c141100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00b3c0 tx=0x7fd93c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c020070 con 0x7fd94c09f140 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: dump: 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: _finish_auth 0 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd95126e700 5 monclient: authenticate success, global_id 4406 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c006a90 con 0x7fd94c09f140 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd95126e700 10 monclient: _renew_subs 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd95126e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c11df80 con 0x7fd94c09f140 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c0256a0 con 0x7fd94c09f140 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd933fff700 10 monclient: dump: 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd95126e700 10 monclient: _renew_subs 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd95126e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94c11f150 con 0x7fd94c09f140 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.563+0000 7fd94a7fc700 4 set_mon_vals no callback set 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd95126e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd95126e700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 0x7fd94c11b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd95126e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c064840 con 0x7fd94c1312e0 2024-09-18T14:21:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fd93c01a420 con 0x7fd94c09f140 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 0x7fd94c11b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient: get_auth_request con 0x7fd94c1312e0 auth_method 0 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd933fff700 1 --2- 172.21.15.104:0/798226483 >> v2:172.21.15.174:6812/17603 conn(0x7fd938035a80 0x7fd938037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fd93c049710 con 0x7fd94c09f140 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd950a6d700 1 --2- 172.21.15.104:0/798226483 >> v2:172.21.15.174:6812/17603 conn(0x7fd938035a80 0x7fd938037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd950a6d700 10 monclient: get_auth_request con 0x7fd938035a80 auth_method 0 2024-09-18T14:21:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-09-18T14:21:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.567+0000 7fd94bfff700 1 -- 172.21.15.104:0/798226483 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd94c11d3d0 con 0x7fd94c1312e0 2024-09-18T14:21:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd94bfff700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 msgr2=0x7fd94c11b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd94bfff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 0x7fd94c11b920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd940010950 tx=0x7fd940008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd950a6d700 1 --2- 172.21.15.104:0/798226483 >> v2:172.21.15.174:6812/17603 conn(0x7fd938035a80 0x7fd938037f30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9440023e0 tx=0x7fd944007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000eb70 con 0x7fd94c1312e0 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 10 monclient: dump: 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd94000ecd0 con 0x7fd94c1312e0 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 msgr2=0x7fd94c11b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 0x7fd94c11b920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd940010950 tx=0x7fd940008040 comp rx=0 tx=0).stop 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 msgr2=0x7fd94c11b920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.571+0000 7fd933fff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c1312e0 0x7fd94c11b920 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd940010950 tx=0x7fd940008040 comp rx=0 tx=0).stop 2024-09-18T14:21:57.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd95126e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:21:57.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd95126e700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 0x7fd94c11c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:21:57.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd95126e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:21:57.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c11c770 con 0x7fd94c11be60 2024-09-18T14:21:57.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 0x7fd94c11c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:21:57.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient: get_auth_request con 0x7fd94c11be60 auth_method 0 2024-09-18T14:21:57.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:21:57.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:21:57.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:21:57.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:21:57.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:21:57.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:21:57.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-09-18T14:21:57.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.583+0000 7fd94bfff700 1 -- 172.21.15.104:0/798226483 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd94c063450 con 0x7fd94c11be60 2024-09-18T14:21:57.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd94bfff700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 msgr2=0x7fd94c11c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:21:57.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd94bfff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 0x7fd94c11c230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd940011010 tx=0x7fd94001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:21:57.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94001eb70 con 0x7fd94c11be60 2024-09-18T14:21:57.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:21:57.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:21:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 10 monclient: dump: 2024-09-18T14:21:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:21:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:21:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:21:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:21:57.599 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:21:57.600 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:21:57.600 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:21:57.600 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:21:57.600 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:21:57.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7fd94001ecd0 con 0x7fd94c11be60 2024-09-18T14:21:57.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:21:57.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:21:57.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 msgr2=0x7fd94c11c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 0x7fd94c11c230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd940011010 tx=0x7fd94001c000 comp rx=0 tx=0).stop 2024-09-18T14:21:57.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 msgr2=0x7fd94c11c230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:21:57.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd933fff700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd94c11be60 0x7fd94c11c230 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd940011010 tx=0x7fd94001c000 comp rx=0 tx=0).stop 2024-09-18T14:21:57.602 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:21:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:21:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:21:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:21:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 12, 2024-09-18T14:21:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:21:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:21:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:21:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 1, 2024-09-18T14:21:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:21:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:21:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:21:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:21:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:21:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:21:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:21:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:21:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:21:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:21:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:21:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:21:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:21:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:21:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:21:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:21:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:21:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:21:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:21:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:21:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:21:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:21:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:21:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:21:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:21:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:21:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:21:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:21:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:21:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:21:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:21:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:21:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:21:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:21:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:21:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:21:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:21:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:21:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:21:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:21:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 6 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:21:57.624 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:21:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 >> v2:172.21.15.174:6812/17603 conn(0x7fd938035a80 msgr2=0x7fd938037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 --2- 172.21.15.104:0/798226483 >> v2:172.21.15.174:6812/17603 conn(0x7fd938035a80 0x7fd938037f30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9440023e0 tx=0x7fd944007e80 comp rx=0 tx=0).stop 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 10 monclient: shutdown 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 msgr2=0x7fd94c141100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 0x7fd94c141100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00b3c0 tx=0x7fd93c017000 comp rx=0 tx=0).stop 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd950a6d700 1 -- 172.21.15.104:0/798226483 reap_dead start 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 shutdown_connections 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 --2- 172.21.15.104:0/798226483 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fd94c09f140 0x7fd94c141100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 >> 172.21.15.104:0/798226483 conn(0x7fd94c0676a0 msgr2=0x7fd94c09cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 shutdown_connections 2024-09-18T14:21:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:21:57.587+0000 7fd95126e700 1 -- 172.21.15.104:0/798226483 wait complete. 2024-09-18T14:21:57.627 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T14:21:57.627 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T14:21:57.666 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18276]: starting. 2024-09-18T14:21:57.666 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18276]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18274 2024-09-18T14:21:57.666 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18276]: shutting down. 2024-09-18T14:21:59.861 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:21:59.861 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18310]: starting. 2024-09-18T14:21:59.862 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18310]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18274 2024-09-18T14:21:59.862 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18310]: created object 0... 2024-09-18T14:21:59.862 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18310]: created object 25... 2024-09-18T14:21:59.862 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18310]: created object 49... 2024-09-18T14:21:59.862 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18310]: finishing. 2024-09-18T14:21:59.862 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18310]: shutting down. 2024-09-18T14:21:59.923 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:21:59.923 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18312]: starting. 2024-09-18T14:21:59.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18312]: listing objects. 2024-09-18T14:21:59.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18312]: listed object 0... 2024-09-18T14:21:59.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18312]: listed object 25... 2024-09-18T14:21:59.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18312]: saw 50 objects 2024-09-18T14:21:59.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18312]: shutting down. 2024-09-18T14:22:01.705 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:01.705 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18311]: starting. 2024-09-18T14:22:01.705 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18311]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18274 2024-09-18T14:22:01.705 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18311]: shutting down. 2024-09-18T14:22:01.709 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:01.710 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:01.710 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:01.710 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:03.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18365]: starting. 2024-09-18T14:22:03.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18365]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18364 2024-09-18T14:22:03.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18365]: created object 0... 2024-09-18T14:22:03.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18365]: created object 25... 2024-09-18T14:22:03.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18365]: created object 49... 2024-09-18T14:22:03.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18365]: finishing. 2024-09-18T14:22:03.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18365]: shutting down. 2024-09-18T14:22:04.721 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18366]: starting. 2024-09-18T14:22:04.721 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18366]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18364 2024-09-18T14:22:04.721 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18366]: shutting down. 2024-09-18T14:22:06.938 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:06.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18401]: starting. 2024-09-18T14:22:06.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18401]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18364 2024-09-18T14:22:06.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18401]: created object 0... 2024-09-18T14:22:06.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18401]: created object 25... 2024-09-18T14:22:06.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18401]: created object 49... 2024-09-18T14:22:06.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18401]: finishing. 2024-09-18T14:22:06.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18401]: shutting down. 2024-09-18T14:22:07.001 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:07.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18403]: starting. 2024-09-18T14:22:07.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18403]: listing objects. 2024-09-18T14:22:07.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18403]: listed object 0... 2024-09-18T14:22:07.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18403]: listed object 25... 2024-09-18T14:22:07.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18403]: saw 50 objects 2024-09-18T14:22:07.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18403]: shutting down. 2024-09-18T14:22:07.753 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:07.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18402]: starting. 2024-09-18T14:22:07.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18402]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18364 2024-09-18T14:22:07.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18402]: shutting down. 2024-09-18T14:22:07.754 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:07.754 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:07.754 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:07.756 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18456]: starting. 2024-09-18T14:22:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18456]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18455 2024-09-18T14:22:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18456]: created object 0... 2024-09-18T14:22:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18456]: created object 25... 2024-09-18T14:22:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18456]: created object 49... 2024-09-18T14:22:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18456]: finishing. 2024-09-18T14:22:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18456]: shutting down. 2024-09-18T14:22:10.784 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18457]: starting. 2024-09-18T14:22:10.784 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18457]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18455 2024-09-18T14:22:10.784 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18457]: shutting down. 2024-09-18T14:22:13.397 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:13.397 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18492]: starting. 2024-09-18T14:22:13.398 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18492]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18455 2024-09-18T14:22:13.398 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18492]: created object 0... 2024-09-18T14:22:13.398 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18492]: created object 25... 2024-09-18T14:22:13.398 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18492]: created object 49... 2024-09-18T14:22:13.398 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18492]: finishing. 2024-09-18T14:22:13.398 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18492]: shutting down. 2024-09-18T14:22:13.451 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:13.452 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18494]: starting. 2024-09-18T14:22:13.452 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18494]: listing objects. 2024-09-18T14:22:13.452 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18494]: listed object 0... 2024-09-18T14:22:13.452 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18494]: listed object 25... 2024-09-18T14:22:13.452 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18494]: saw 50 objects 2024-09-18T14:22:13.452 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18494]: shutting down. 2024-09-18T14:22:13.811 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:13.811 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18493]: starting. 2024-09-18T14:22:13.811 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18493]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18455 2024-09-18T14:22:13.811 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18493]: shutting down. 2024-09-18T14:22:13.813 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:13.813 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:13.813 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:13.814 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:16.025 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18547]: starting. 2024-09-18T14:22:16.025 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18547]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18546 2024-09-18T14:22:16.025 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18547]: created object 0... 2024-09-18T14:22:16.025 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18547]: created object 25... 2024-09-18T14:22:16.025 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18547]: created object 49... 2024-09-18T14:22:16.026 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18547]: finishing. 2024-09-18T14:22:16.026 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18547]: shutting down. 2024-09-18T14:22:16.848 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18548]: starting. 2024-09-18T14:22:16.849 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18548]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18546 2024-09-18T14:22:16.849 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18548]: shutting down. 2024-09-18T14:22:18.667 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:18.668 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18583]: starting. 2024-09-18T14:22:18.668 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18583]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18546 2024-09-18T14:22:18.668 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18583]: created object 0... 2024-09-18T14:22:18.668 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18583]: created object 25... 2024-09-18T14:22:18.668 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18583]: created object 49... 2024-09-18T14:22:18.668 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18583]: finishing. 2024-09-18T14:22:18.668 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18583]: shutting down. 2024-09-18T14:22:18.727 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:18.727 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18585]: starting. 2024-09-18T14:22:18.727 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18585]: listing objects. 2024-09-18T14:22:18.727 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18585]: listed object 0... 2024-09-18T14:22:18.728 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18585]: listed object 25... 2024-09-18T14:22:18.728 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18585]: saw 50 objects 2024-09-18T14:22:18.728 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18585]: shutting down. 2024-09-18T14:22:19.524 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:19.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18584]: starting. 2024-09-18T14:22:19.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18584]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18546 2024-09-18T14:22:19.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18584]: shutting down. 2024-09-18T14:22:19.525 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:19.525 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:19.525 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:19.527 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:22.331 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18638]: starting. 2024-09-18T14:22:22.332 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18638]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18637 2024-09-18T14:22:22.332 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18638]: created object 0... 2024-09-18T14:22:22.332 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18638]: created object 25... 2024-09-18T14:22:22.332 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18638]: created object 49... 2024-09-18T14:22:22.332 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18638]: finishing. 2024-09-18T14:22:22.332 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18638]: shutting down. 2024-09-18T14:22:22.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18639]: starting. 2024-09-18T14:22:22.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18639]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18637 2024-09-18T14:22:22.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18639]: shutting down. 2024-09-18T14:22:24.701 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:24.701 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18674]: starting. 2024-09-18T14:22:24.701 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18674]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18637 2024-09-18T14:22:24.701 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18674]: created object 0... 2024-09-18T14:22:24.702 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18674]: created object 25... 2024-09-18T14:22:24.702 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18674]: created object 49... 2024-09-18T14:22:24.702 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18674]: finishing. 2024-09-18T14:22:24.702 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18674]: shutting down. 2024-09-18T14:22:24.763 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:24.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18676]: starting. 2024-09-18T14:22:24.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18676]: listing objects. 2024-09-18T14:22:24.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18676]: listed object 0... 2024-09-18T14:22:24.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18676]: listed object 25... 2024-09-18T14:22:24.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18676]: saw 50 objects 2024-09-18T14:22:24.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18676]: shutting down. 2024-09-18T14:22:25.530 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:25.530 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18675]: starting. 2024-09-18T14:22:25.531 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18675]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18637 2024-09-18T14:22:25.531 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18675]: shutting down. 2024-09-18T14:22:25.532 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:25.532 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:25.532 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:25.534 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18729]: starting. 2024-09-18T14:22:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18729]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18728 2024-09-18T14:22:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18729]: created object 0... 2024-09-18T14:22:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18729]: created object 25... 2024-09-18T14:22:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18729]: created object 49... 2024-09-18T14:22:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18729]: finishing. 2024-09-18T14:22:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18729]: shutting down. 2024-09-18T14:22:28.563 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18730]: starting. 2024-09-18T14:22:28.563 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18730]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18728 2024-09-18T14:22:28.563 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18730]: shutting down. 2024-09-18T14:22:30.741 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:30.741 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18765]: starting. 2024-09-18T14:22:30.741 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18765]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18728 2024-09-18T14:22:30.741 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18765]: created object 0... 2024-09-18T14:22:30.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18765]: created object 25... 2024-09-18T14:22:30.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18765]: created object 49... 2024-09-18T14:22:30.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18765]: finishing. 2024-09-18T14:22:30.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18765]: shutting down. 2024-09-18T14:22:30.803 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:30.803 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18767]: starting. 2024-09-18T14:22:30.803 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18767]: listing objects. 2024-09-18T14:22:30.803 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18767]: listed object 0... 2024-09-18T14:22:30.804 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18767]: listed object 25... 2024-09-18T14:22:30.804 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18767]: saw 50 objects 2024-09-18T14:22:30.804 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18767]: shutting down. 2024-09-18T14:22:32.613 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:32.614 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18766]: starting. 2024-09-18T14:22:32.614 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18766]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18728 2024-09-18T14:22:32.614 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18766]: shutting down. 2024-09-18T14:22:32.614 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:32.614 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:32.614 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:32.615 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:34.717 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18820]: starting. 2024-09-18T14:22:34.717 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18820]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18819 2024-09-18T14:22:34.717 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18820]: created object 0... 2024-09-18T14:22:34.717 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18820]: created object 25... 2024-09-18T14:22:34.717 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18820]: created object 49... 2024-09-18T14:22:34.717 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18820]: finishing. 2024-09-18T14:22:34.717 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18820]: shutting down. 2024-09-18T14:22:35.532 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18821]: starting. 2024-09-18T14:22:35.532 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18821]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18819 2024-09-18T14:22:35.532 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18821]: shutting down. 2024-09-18T14:22:37.709 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:37.710 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18856]: starting. 2024-09-18T14:22:37.710 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18856]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18819 2024-09-18T14:22:37.710 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18856]: created object 0... 2024-09-18T14:22:37.710 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18856]: created object 25... 2024-09-18T14:22:37.710 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18856]: created object 49... 2024-09-18T14:22:37.710 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18856]: finishing. 2024-09-18T14:22:37.710 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18856]: shutting down. 2024-09-18T14:22:37.770 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:37.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18858]: starting. 2024-09-18T14:22:37.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18858]: listing objects. 2024-09-18T14:22:37.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18858]: listed object 0... 2024-09-18T14:22:37.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18858]: listed object 25... 2024-09-18T14:22:37.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18858]: saw 50 objects 2024-09-18T14:22:37.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18858]: shutting down. 2024-09-18T14:22:38.559 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:38.559 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18857]: starting. 2024-09-18T14:22:38.559 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18857]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18819 2024-09-18T14:22:38.559 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18857]: shutting down. 2024-09-18T14:22:38.560 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:38.560 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:38.561 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:38.562 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18911]: starting. 2024-09-18T14:22:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18911]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18910 2024-09-18T14:22:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18911]: created object 0... 2024-09-18T14:22:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18911]: created object 25... 2024-09-18T14:22:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18911]: created object 49... 2024-09-18T14:22:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18911]: finishing. 2024-09-18T14:22:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[18911]: shutting down. 2024-09-18T14:22:41.592 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18912]: starting. 2024-09-18T14:22:41.592 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18912]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18910 2024-09-18T14:22:41.592 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[18912]: shutting down. 2024-09-18T14:22:43.691 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:43.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18947]: starting. 2024-09-18T14:22:43.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18947]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-18910 2024-09-18T14:22:43.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18947]: created object 0... 2024-09-18T14:22:43.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18947]: created object 25... 2024-09-18T14:22:43.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18947]: created object 49... 2024-09-18T14:22:43.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18947]: finishing. 2024-09-18T14:22:43.692 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[18947]: shutting down. 2024-09-18T14:22:43.752 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:43.752 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18949]: starting. 2024-09-18T14:22:43.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18949]: listing objects. 2024-09-18T14:22:43.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18949]: listed object 0... 2024-09-18T14:22:43.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18949]: listed object 25... 2024-09-18T14:22:43.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18949]: saw 50 objects 2024-09-18T14:22:43.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[18949]: shutting down. 2024-09-18T14:22:44.533 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:44.533 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18948]: starting. 2024-09-18T14:22:44.533 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18948]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-18910 2024-09-18T14:22:44.533 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[18948]: shutting down. 2024-09-18T14:22:44.534 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:44.534 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:44.534 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:44.536 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:46.879 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19002]: starting. 2024-09-18T14:22:46.879 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19002]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19001 2024-09-18T14:22:46.879 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19002]: created object 0... 2024-09-18T14:22:46.879 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19002]: created object 25... 2024-09-18T14:22:46.879 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19002]: created object 49... 2024-09-18T14:22:46.879 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19002]: finishing. 2024-09-18T14:22:46.880 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19002]: shutting down. 2024-09-18T14:22:47.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19003]: starting. 2024-09-18T14:22:47.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19003]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19001 2024-09-18T14:22:47.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19003]: shutting down. 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19038]: starting. 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19038]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19001 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19038]: created object 0... 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19038]: created object 25... 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19038]: created object 49... 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19038]: finishing. 2024-09-18T14:22:49.885 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19038]: shutting down. 2024-09-18T14:22:49.946 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:49.946 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19040]: starting. 2024-09-18T14:22:49.946 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19040]: listing objects. 2024-09-18T14:22:49.946 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19040]: listed object 0... 2024-09-18T14:22:49.946 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19040]: listed object 25... 2024-09-18T14:22:49.946 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19040]: saw 50 objects 2024-09-18T14:22:49.946 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19040]: shutting down. 2024-09-18T14:22:50.754 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:50.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19039]: starting. 2024-09-18T14:22:50.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19039]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19001 2024-09-18T14:22:50.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19039]: shutting down. 2024-09-18T14:22:50.755 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:50.755 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:50.755 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:50.756 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:53.575 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19093]: starting. 2024-09-18T14:22:53.575 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19093]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19092 2024-09-18T14:22:53.575 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19093]: created object 0... 2024-09-18T14:22:53.575 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19093]: created object 25... 2024-09-18T14:22:53.575 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19093]: created object 49... 2024-09-18T14:22:53.575 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19093]: finishing. 2024-09-18T14:22:53.575 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19093]: shutting down. 2024-09-18T14:22:54.538 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19094]: starting. 2024-09-18T14:22:54.538 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19094]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19092 2024-09-18T14:22:54.538 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19094]: shutting down. 2024-09-18T14:22:56.708 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:56.708 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19129]: starting. 2024-09-18T14:22:56.708 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19129]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19092 2024-09-18T14:22:56.708 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19129]: created object 0... 2024-09-18T14:22:56.708 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19129]: created object 25... 2024-09-18T14:22:56.708 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19129]: created object 49... 2024-09-18T14:22:56.709 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19129]: finishing. 2024-09-18T14:22:56.709 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19129]: shutting down. 2024-09-18T14:22:56.769 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:56.769 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19131]: starting. 2024-09-18T14:22:56.769 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19131]: listing objects. 2024-09-18T14:22:56.769 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19131]: listed object 0... 2024-09-18T14:22:56.769 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19131]: listed object 25... 2024-09-18T14:22:56.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19131]: saw 50 objects 2024-09-18T14:22:56.770 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19131]: shutting down. 2024-09-18T14:22:57.554 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:57.554 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19130]: starting. 2024-09-18T14:22:57.555 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19130]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19092 2024-09-18T14:22:57.555 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19130]: shutting down. 2024-09-18T14:22:57.555 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:57.556 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:22:57.556 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:22:57.557 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:22:59.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19184]: starting. 2024-09-18T14:22:59.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19184]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19183 2024-09-18T14:22:59.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19184]: created object 0... 2024-09-18T14:22:59.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19184]: created object 25... 2024-09-18T14:22:59.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19184]: created object 49... 2024-09-18T14:22:59.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19184]: finishing. 2024-09-18T14:22:59.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19184]: shutting down. 2024-09-18T14:23:00.579 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19185]: starting. 2024-09-18T14:23:00.579 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19185]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19183 2024-09-18T14:23:00.579 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19185]: shutting down. 2024-09-18T14:23:02.813 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:02.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19220]: starting. 2024-09-18T14:23:02.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19220]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19183 2024-09-18T14:23:02.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19220]: created object 0... 2024-09-18T14:23:02.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19220]: created object 25... 2024-09-18T14:23:02.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19220]: created object 49... 2024-09-18T14:23:02.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19220]: finishing. 2024-09-18T14:23:02.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19220]: shutting down. 2024-09-18T14:23:03.713 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:03.713 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19222]: starting. 2024-09-18T14:23:03.713 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19222]: listing objects. 2024-09-18T14:23:03.713 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19222]: listed object 0... 2024-09-18T14:23:03.713 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19222]: listed object 25... 2024-09-18T14:23:03.713 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19222]: saw 50 objects 2024-09-18T14:23:03.713 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19222]: shutting down. 2024-09-18T14:23:05.322 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:05.322 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19221]: starting. 2024-09-18T14:23:05.322 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19221]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19183 2024-09-18T14:23:05.322 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19221]: shutting down. 2024-09-18T14:23:05.323 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:05.323 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:05.323 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:23:05.325 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:23:07.523 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19275]: starting. 2024-09-18T14:23:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19275]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19274 2024-09-18T14:23:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19275]: created object 0... 2024-09-18T14:23:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19275]: created object 25... 2024-09-18T14:23:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19275]: created object 49... 2024-09-18T14:23:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19275]: finishing. 2024-09-18T14:23:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19275]: shutting down. 2024-09-18T14:23:08.343 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19276]: starting. 2024-09-18T14:23:08.343 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19276]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19274 2024-09-18T14:23:08.343 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19276]: shutting down. 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19311]: starting. 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19311]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19274 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19311]: created object 0... 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19311]: created object 25... 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19311]: created object 49... 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19311]: finishing. 2024-09-18T14:23:10.540 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19311]: shutting down. 2024-09-18T14:23:10.599 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:10.599 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19313]: starting. 2024-09-18T14:23:10.599 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19313]: listing objects. 2024-09-18T14:23:10.599 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19313]: listed object 0... 2024-09-18T14:23:10.600 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19313]: listed object 25... 2024-09-18T14:23:10.600 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19313]: saw 50 objects 2024-09-18T14:23:10.600 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19313]: shutting down. 2024-09-18T14:23:11.378 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:11.378 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19312]: starting. 2024-09-18T14:23:11.379 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19312]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19274 2024-09-18T14:23:11.379 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19312]: shutting down. 2024-09-18T14:23:11.379 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:11.379 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:11.379 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:23:11.381 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:23:13.543 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19366]: starting. 2024-09-18T14:23:13.543 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19366]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19365 2024-09-18T14:23:13.543 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19366]: created object 0... 2024-09-18T14:23:13.543 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19366]: created object 25... 2024-09-18T14:23:13.543 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19366]: created object 49... 2024-09-18T14:23:13.543 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19366]: finishing. 2024-09-18T14:23:13.543 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19366]: shutting down. 2024-09-18T14:23:14.545 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19367]: starting. 2024-09-18T14:23:14.545 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19367]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19365 2024-09-18T14:23:14.545 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19367]: shutting down. 2024-09-18T14:23:16.755 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:16.756 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19402]: starting. 2024-09-18T14:23:16.756 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19402]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19365 2024-09-18T14:23:16.756 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19402]: created object 0... 2024-09-18T14:23:16.756 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19402]: created object 25... 2024-09-18T14:23:16.756 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19402]: created object 49... 2024-09-18T14:23:16.756 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19402]: finishing. 2024-09-18T14:23:16.756 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19402]: shutting down. 2024-09-18T14:23:16.815 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:16.815 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19404]: starting. 2024-09-18T14:23:16.815 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19404]: listing objects. 2024-09-18T14:23:16.815 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19404]: listed object 0... 2024-09-18T14:23:16.815 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19404]: listed object 25... 2024-09-18T14:23:16.815 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19404]: saw 50 objects 2024-09-18T14:23:16.815 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19404]: shutting down. 2024-09-18T14:23:17.564 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:17.564 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19403]: starting. 2024-09-18T14:23:17.564 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19403]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19365 2024-09-18T14:23:17.564 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19403]: shutting down. 2024-09-18T14:23:17.565 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:17.565 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:17.565 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:23:17.567 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:23:19.743 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19457]: starting. 2024-09-18T14:23:19.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19457]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19456 2024-09-18T14:23:19.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19457]: created object 0... 2024-09-18T14:23:19.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19457]: created object 25... 2024-09-18T14:23:19.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19457]: created object 49... 2024-09-18T14:23:19.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19457]: finishing. 2024-09-18T14:23:19.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19457]: shutting down. 2024-09-18T14:23:20.602 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19458]: starting. 2024-09-18T14:23:20.602 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19458]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19456 2024-09-18T14:23:20.602 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19458]: shutting down. 2024-09-18T14:23:22.777 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:22.777 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19495]: starting. 2024-09-18T14:23:22.777 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19495]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19456 2024-09-18T14:23:22.777 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19495]: created object 0... 2024-09-18T14:23:22.777 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19495]: created object 25... 2024-09-18T14:23:22.777 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19495]: created object 49... 2024-09-18T14:23:22.777 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19495]: finishing. 2024-09-18T14:23:22.778 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19495]: shutting down. 2024-09-18T14:23:22.834 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:22.834 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19497]: starting. 2024-09-18T14:23:22.834 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19497]: listing objects. 2024-09-18T14:23:22.835 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19497]: listed object 0... 2024-09-18T14:23:22.835 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19497]: listed object 25... 2024-09-18T14:23:22.835 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19497]: saw 50 objects 2024-09-18T14:23:22.835 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19497]: shutting down. 2024-09-18T14:23:23.542 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:23.542 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19496]: starting. 2024-09-18T14:23:23.542 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19496]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19456 2024-09-18T14:23:23.542 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19496]: shutting down. 2024-09-18T14:23:23.543 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:23.543 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:23.543 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:23:23.545 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:23:25.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19550]: starting. 2024-09-18T14:23:25.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19550]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19549 2024-09-18T14:23:25.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19550]: created object 0... 2024-09-18T14:23:25.691 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19550]: created object 25... 2024-09-18T14:23:25.692 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19550]: created object 49... 2024-09-18T14:23:25.692 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19550]: finishing. 2024-09-18T14:23:25.692 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19550]: shutting down. 2024-09-18T14:23:26.555 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19551]: starting. 2024-09-18T14:23:26.555 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19551]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19549 2024-09-18T14:23:26.555 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19551]: shutting down. 2024-09-18T14:23:27.627 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T14:23:27.628 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T14:23:27.628 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T14:23:27.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T14:23:27.631 INFO:tasks.ceph.mon.a:Started 2024-09-18T14:23:27.631 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T14:23:27.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:27.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient: get_monmap_and_config 2024-09-18T14:23:27.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient: build_initial_monmap 2024-09-18T14:23:27.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient: monmap: 2024-09-18T14:23:27.736 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:27.736 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:27.736028+0000 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:27.736028+0000 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:27.737 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:27.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 1 Processor -- start 2024-09-18T14:23:27.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 1 -- start start 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient: init 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcdc006ee40 0x7fcdc012fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient(hunting): picked mon.noname-c con 0x7fcdc006ee40 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdc012ff80 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 0x7fcdc009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient(hunting): picked mon.noname-a con 0x7fcdc0130380 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc006f210 con 0x7fcdc012ff80 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc006e080 con 0x7fcdc0130380 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc006e360 con 0x7fcdc006ee40 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc5268700 10 monclient(hunting): authenticate will time out at 928.725622s 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 0x7fcdc009e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 0x7fcdc009e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59070/0 (socket says 172.21.15.104:59070) 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 1 -- 172.21.15.104:0/3217863844 learned_addr learned my addr 172.21.15.104:0/3217863844 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 10 monclient(hunting): get_auth_request con 0x7fcdc0130380 auth_method 0 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 1 --2- 172.21.15.104:0/3217863844 >> v2:172.21.15.174:3300/0 conn(0x7fcdc006ee40 0x7fcdc012fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 1 -- 172.21.15.104:0/3217863844 >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 msgr2=0x7fcdc009e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 1 -- 172.21.15.104:0/3217863844 >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 msgr2=0x7fcdc009e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): get_auth_request con 0x7fcdc006ee40 auth_method 0 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 1 --2- 172.21.15.104:0/3217863844 >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 0x7fcdc009e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdc4a67700 1 --2- 172.21.15.104:0/3217863844 >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 0x7fcdc009e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:27.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient(hunting): handle_auth_done global_id 4765 payload 274 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 1 -- 172.21.15.104:0/3217863844 >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 msgr2=0x7fcdc009e6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 1 --2- 172.21.15.104:0/3217863844 >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 0x7fcdc009e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 1 -- 172.21.15.104:0/3217863844 >> v1:172.21.15.174:6789/0 conn(0x7fcdc012ff80 legacy=0x7fcdc009dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient: _finish_hunting 0 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 1 monclient: found mon.noname-c 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:23:27.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.733+0000 7fcdbffff700 1 -- 172.21.15.104:0/3217863844 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc006e640 con 0x7fcdc006ee40 2024-09-18T14:23:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 1 -- 172.21.15.104:0/3217863844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc006ee40 msgr2=0x7fcdc012fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 1 --2- 172.21.15.104:0/3217863844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc006ee40 0x7fcdc012fa40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcdc006cbf0 tx=0x7fcdb00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c47d1a04d22305 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 1 -- 172.21.15.104:0/3217863844 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000b8c0 con 0x7fcdc006ee40 2024-09-18T14:23:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: dump: 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:27.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: _finish_auth 0 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 1 -- 172.21.15.104:0/3217863844 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb000ba20 con 0x7fcdc006ee40 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 5 monclient: authenticate success, global_id 4765 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 1 -- 172.21.15.104:0/3217863844 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000bdc0 con 0x7fcdc006ee40 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 4 set_mon_vals no callback set 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbeffd700 10 monclient: dump: 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:27.750 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient: shutdown 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- 172.21.15.104:0/3217863844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc006ee40 msgr2=0x7fcdc012fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 --2- 172.21.15.104:0/3217863844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc006ee40 0x7fcdc012fa40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcdc006cbf0 tx=0x7fcdb00091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- 172.21.15.104:0/3217863844 shutdown_connections 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 --2- 172.21.15.104:0/3217863844 >> v2:172.21.15.104:3300/0 conn(0x7fcdc0130380 0x7fcdc009e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 --2- 172.21.15.104:0/3217863844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc006ee40 0x7fcdc012fa40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- 172.21.15.104:0/3217863844 >> 172.21.15.104:0/3217863844 conn(0x7fcdc00682c0 msgr2=0x7fcdc00a30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- 172.21.15.104:0/3217863844 shutdown_connections 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- 172.21.15.104:0/3217863844 wait complete. 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient: build_initial_monmap 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient: monmap: 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:27.752 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 Processor -- start 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- start start 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient: init 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 0x7fcdc0098b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): picked mon.noname-c con 0x7fcdc006ee40 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 0x7fcdc012eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): picked mon.noname-a con 0x7fcdc012ff80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc0130380 0x7fcdc012f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:27.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): picked mon.noname-b con 0x7fcdc0130380 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc011c060 con 0x7fcdc012ff80 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc009c7b0 con 0x7fcdc0130380 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc011a6f0 con 0x7fcdc006ee40 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdc5268700 10 monclient(hunting): authenticate will time out at 928.731707s 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 0x7fcdc0098b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 0x7fcdc0098b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:40768/0 (socket says 172.21.15.104:40768) 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 1 -- 172.21.15.104:0/1198519808 learned_addr learned my addr 172.21.15.104:0/1198519808 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:27.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 10 monclient(hunting): get_auth_request con 0x7fcdc006ee40 auth_method 0 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.737+0000 7fcdbf7fe700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 0x7fcdc012eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc0130380 0x7fcdc012f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 10 monclient(hunting): get_auth_request con 0x7fcdc012ff80 auth_method 0 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:27.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 10 monclient(hunting): get_auth_request con 0x7fcdc0130380 auth_method 0 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 1 -- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 msgr2=0x7fcdc012eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 1 -- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 msgr2=0x7fcdc012eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc4a67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 0x7fcdc012eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 10 monclient(hunting): handle_auth_done global_id 4751 payload 824 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbf7fe700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 0x7fcdc012eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:27.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 -- 172.21.15.104:0/1198519808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc0130380 msgr2=0x7fcdc012f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc0130380 0x7fcdc012f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 -- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 msgr2=0x7fcdc012eb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 0x7fcdc012eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 10 monclient: _finish_hunting 0 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 monclient: found mon.noname-c 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 -- 172.21.15.104:0/1198519808 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb0009420 con 0x7fcdc006ee40 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 -- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 msgr2=0x7fcdc0098b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbffff700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 0x7fcdc0098b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000b0e0 tx=0x7fcdb0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 1 -- 172.21.15.104:0/1198519808 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0011bb0 con 0x7fcdc006ee40 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:27.757 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: dump: 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: _finish_auth 0 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:27.758 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc5268700 5 monclient: authenticate success, global_id 4751 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc5268700 10 monclient: _renew_subs 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 1 -- 172.21.15.104:0/1198519808 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb0020070 con 0x7fcdc006ee40 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc5268700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc006d3d0 con 0x7fcdc006ee40 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 1 -- 172.21.15.104:0/1198519808 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb001b450 con 0x7fcdc006ee40 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcda7fff700 10 monclient: dump: 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:27.759 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdbe7fc700 4 set_mon_vals no callback set 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc5268700 10 monclient: _renew_subs 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc5268700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:27.760 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.741+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc011ad90 con 0x7fcdc006ee40 2024-09-18T14:23:27.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcdc5268700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:27.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcdc5268700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:27.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcdc5268700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:27.761 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcda7fff700 1 -- 172.21.15.104:0/1198519808 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fcdb0022030 con 0x7fcdc006ee40 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc011c340 con 0x7fcdc006ee40 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcda7fff700 1 --2- 172.21.15.104:0/1198519808 >> v2:172.21.15.174:6812/17603 conn(0x7fcdac035a80 0x7fcdac037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcda7fff700 1 -- 172.21.15.104:0/1198519808 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fcdb0049c60 con 0x7fcdc006ee40 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcdbf7fe700 1 --2- 172.21.15.104:0/1198519808 >> v2:172.21.15.174:6812/17603 conn(0x7fcdac035a80 0x7fcdac037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.745+0000 7fcdbf7fe700 10 monclient: get_auth_request con 0x7fcdac035a80 auth_method 0 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.749+0000 7fcdbf7fe700 1 --2- 172.21.15.104:0/1198519808 >> v2:172.21.15.174:6812/17603 conn(0x7fcdac035a80 0x7fcdac037f30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdc012f8b0 tx=0x7fcdb4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.753+0000 7fcda7fff700 1 -- 172.21.15.104:0/1198519808 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcdb001b800 con 0x7fcdc006ee40 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.753+0000 7fcda7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:27.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.753+0000 7fcda7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.977+0000 7fcdc5268700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:27.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.977+0000 7fcdc5268700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:27.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.977+0000 7fcdc5268700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:27.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.977+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcdc011c340 con 0x7fcdc006ee40 2024-09-18T14:23:27.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcda7fff700 1 -- 172.21.15.104:0/1198519808 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fcdb001a3d0 con 0x7fcdc006ee40 2024-09-18T14:23:27.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcda7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:27.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcda7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:27.985 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 >> v2:172.21.15.174:6812/17603 conn(0x7fcdac035a80 msgr2=0x7fcdac037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 --2- 172.21.15.104:0/1198519808 >> v2:172.21.15.174:6812/17603 conn(0x7fcdac035a80 0x7fcdac037f30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdc012f8b0 tx=0x7fcdb4009040 comp rx=0 tx=0).stop 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 10 monclient: shutdown 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 msgr2=0x7fcdc0098b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 0x7fcdc0098b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000b0e0 tx=0x7fcdb0016000 comp rx=0 tx=0).stop 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 shutdown_connections 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 --2- 172.21.15.104:0/1198519808 >> v2:172.21.15.174:6812/17603 conn(0x7fcdac035a80 0x7fcdac037f30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc0130380 0x7fcdc012f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc012ff80 0x7fcdc012eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 --2- 172.21.15.104:0/1198519808 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fcdc006ee40 0x7fcdc0098b30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:27.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 >> 172.21.15.104:0/1198519808 conn(0x7fcdc00682c0 msgr2=0x7fcdc0132c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:27.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 shutdown_connections 2024-09-18T14:23:27.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:27.981+0000 7fcdc5268700 1 -- 172.21.15.104:0/1198519808 wait complete. 2024-09-18T14:23:28.722 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:28.722 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19586]: starting. 2024-09-18T14:23:28.722 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19586]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19549 2024-09-18T14:23:28.722 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19586]: created object 0... 2024-09-18T14:23:28.722 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19586]: created object 25... 2024-09-18T14:23:28.723 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19586]: created object 49... 2024-09-18T14:23:28.723 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19586]: finishing. 2024-09-18T14:23:28.723 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19586]: shutting down. 2024-09-18T14:23:28.782 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:28.782 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19588]: starting. 2024-09-18T14:23:28.782 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19588]: listing objects. 2024-09-18T14:23:28.782 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19588]: listed object 0... 2024-09-18T14:23:28.782 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19588]: listed object 25... 2024-09-18T14:23:28.783 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19588]: saw 50 objects 2024-09-18T14:23:28.783 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19588]: shutting down. 2024-09-18T14:23:29.579 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:29.579 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19587]: starting. 2024-09-18T14:23:29.579 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19587]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19549 2024-09-18T14:23:29.579 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19587]: shutting down. 2024-09-18T14:23:29.580 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:29.580 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:29.580 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:23:29.581 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:23:30.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:31.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient: get_monmap_and_config 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient: build_initial_monmap 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient: monmap: 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:31.111902+0000 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:31.111902+0000 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 1 Processor -- start 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 1 -- start start 2024-09-18T14:23:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient: init 2024-09-18T14:23:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 0x7f04bc09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient(hunting): picked mon.noname-a con 0x7f04bc0a59d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f04bc09bea0 0x7f04bc09f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient(hunting): picked mon.noname-c con 0x7f04bc09bea0 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04bc09fcb0 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc09f780 con 0x7f04bc09fcb0 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04bc09af10 con 0x7f04bc0a59d0 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04bc0a1870 con 0x7f04bc09bea0 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04c08d1700 10 monclient(hunting): authenticate will time out at 932.101508s 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 0x7f04bc09ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bb59e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f04bc09fcb0 0x7f04bc0a00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34438/0 (socket says 172.21.15.104:34438) 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 0x7f04bc09ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39578/0 (socket says 172.21.15.104:39578) 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bb59e700 1 -- 172.21.15.104:0/4261684069 learned_addr learned my addr 172.21.15.104:0/4261684069 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 10 monclient(hunting): get_auth_request con 0x7f04bc0a59d0 auth_method 0 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 1 -- 172.21.15.104:0/4261684069 >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 msgr2=0x7f04bc09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 1 -- 172.21.15.104:0/4261684069 >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 msgr2=0x7f04bc09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 1 --2- 172.21.15.104:0/4261684069 >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 0x7f04bc09ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 1 --2- 172.21.15.104:0/4261684069 >> v2:172.21.15.174:3300/0 conn(0x7f04bc09bea0 0x7f04bc09f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): get_auth_request con 0x7f04bc09bea0 auth_method 0 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04bad9d700 1 --2- 172.21.15.104:0/4261684069 >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 0x7f04bc09ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient(hunting): handle_auth_done global_id 4783 payload 274 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 1 -- 172.21.15.104:0/4261684069 >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 msgr2=0x7f04bc09ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 1 --2- 172.21.15.104:0/4261684069 >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 0x7f04bc09ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 1 -- 172.21.15.104:0/4261684069 >> v1:172.21.15.104:6789/0 conn(0x7f04bc09fcb0 legacy=0x7f04bc0a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient: _finish_hunting 0 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 1 monclient: found mon.noname-c 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.109+0000 7f04ba59c700 1 -- 172.21.15.104:0/4261684069 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc0a5da0 con 0x7f04bc09bea0 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04ba59c700 1 -- 172.21.15.104:0/4261684069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09bea0 msgr2=0x7f04bc09f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04ba59c700 1 --2- 172.21.15.104:0/4261684069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09bea0 0x7f04bc09f240 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f04bc06f350 tx=0x7f04b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c00c66d8dc39d990 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 1 -- 172.21.15.104:0/4261684069 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b000b8c0 con 0x7f04bc09bea0 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: dump: 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.125 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: _finish_auth 0 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 5 monclient: authenticate success, global_id 4783 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 1 -- 172.21.15.104:0/4261684069 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b000ba20 con 0x7f04bc09bea0 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:31.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 4 set_mon_vals no callback set 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 1 -- 172.21.15.104:0/4261684069 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b000bdc0 con 0x7f04bc09bea0 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04b9d9b700 10 monclient: dump: 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:31.127 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 10 monclient: shutdown 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 -- 172.21.15.104:0/4261684069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09bea0 msgr2=0x7f04bc09f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 --2- 172.21.15.104:0/4261684069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09bea0 0x7f04bc09f240 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f04bc06f350 tx=0x7f04b00091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 -- 172.21.15.104:0/4261684069 shutdown_connections 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 --2- 172.21.15.104:0/4261684069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09bea0 0x7f04bc09f240 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 --2- 172.21.15.104:0/4261684069 >> v2:172.21.15.104:3300/0 conn(0x7f04bc0a59d0 0x7f04bc09ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 -- 172.21.15.104:0/4261684069 >> 172.21.15.104:0/4261684069 conn(0x7f04bc067c10 msgr2=0x7f04bc0a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 -- 172.21.15.104:0/4261684069 shutdown_connections 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 -- 172.21.15.104:0/4261684069 wait complete. 2024-09-18T14:23:31.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 10 monclient: build_initial_monmap 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 10 monclient: monmap: 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 Processor -- start 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 1 -- start start 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.113+0000 7f04c08d1700 10 monclient: init 2024-09-18T14:23:31.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 0x7f04bc131e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): picked mon.noname-c con 0x7f04bc09bea0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09fcb0 0x7f04bc1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): picked mon.noname-b con 0x7f04bc09fcb0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 0x7f04bc1416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): picked mon.noname-a con 0x7f04bc0a59d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc141c20 con 0x7f04bc0a59d0 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc06f3c0 con 0x7f04bc09fcb0 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1205a0 con 0x7f04bc09bea0 2024-09-18T14:23:31.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient(hunting): authenticate will time out at 932.108194s 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 0x7f04bc131e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 0x7f04bc1416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 0x7f04bc131e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:39110/0 (socket says 172.21.15.104:39110) 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 -- 172.21.15.104:0/2207106236 learned_addr learned my addr 172.21.15.104:0/2207106236 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): get_auth_request con 0x7f04bc09bea0 auth_method 0 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 10 monclient(hunting): get_auth_request con 0x7f04bc0a59d0 auth_method 0 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:31.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 1 -- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 msgr2=0x7f04bc1416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 1 -- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 msgr2=0x7f04bc1416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 0x7f04bc1416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09fcb0 0x7f04bc1411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 10 monclient(hunting): get_auth_request con 0x7f04bc09fcb0 auth_method 0 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bb59e700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 0x7f04bc1416e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:31.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04ba59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient(hunting): handle_auth_done global_id 4763 payload 824 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 -- 172.21.15.104:0/2207106236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09fcb0 msgr2=0x7f04bc1411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09fcb0 0x7f04bc1411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 -- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 msgr2=0x7f04bc1416e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 0x7f04bc1416e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient: _finish_hunting 0 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 monclient: found mon.noname-c 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:23:31.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 -- 172.21.15.104:0/2207106236 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b0009420 con 0x7f04bc09bea0 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 -- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 msgr2=0x7f04bc131e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04bad9d700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 0x7f04bc131e20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f04b400e0e0 tx=0x7f04b4009930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 1 -- 172.21.15.104:0/2207106236 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4014420 con 0x7f04bc09bea0 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: dump: 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.136 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: _finish_auth 0 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 5 monclient: authenticate success, global_id 4763 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 1 -- 172.21.15.104:0/2207106236 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b4014580 con 0x7f04bc09bea0 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient: _renew_subs 2024-09-18T14:23:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 1 -- 172.21.15.104:0/2207106236 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4014940 con 0x7f04bc09bea0 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc11de30 con 0x7f04bc09bea0 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04a2ffd700 10 monclient: dump: 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:31.138 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04b959a700 4 set_mon_vals no callback set 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient: _renew_subs 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.117+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc120230 con 0x7f04bc09bea0 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04c08d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04c08d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04c08d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc0633b0 con 0x7f04bc09bea0 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04a2ffd700 1 -- 172.21.15.104:0/2207106236 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f04b4014cf0 con 0x7f04bc09bea0 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04a2ffd700 1 --2- 172.21.15.104:0/2207106236 >> v2:172.21.15.174:6812/17603 conn(0x7f04a8035a80 0x7f04a8037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04a2ffd700 1 -- 172.21.15.104:0/2207106236 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f04b4048850 con 0x7f04bc09bea0 2024-09-18T14:23:31.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04ba59c700 1 --2- 172.21.15.104:0/2207106236 >> v2:172.21.15.174:6812/17603 conn(0x7f04a8035a80 0x7f04a8037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:31.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.121+0000 7f04ba59c700 10 monclient: get_auth_request con 0x7f04a8035a80 auth_method 0 2024-09-18T14:23:31.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.125+0000 7f04ba59c700 1 --2- 172.21.15.104:0/2207106236 >> v2:172.21.15.174:6812/17603 conn(0x7f04a8035a80 0x7f04a8037f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f04b00030c0 tx=0x7f04b0016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:31.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.129+0000 7f04a2ffd700 1 -- 172.21.15.104:0/2207106236 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f04b40099a0 con 0x7f04bc09bea0 2024-09-18T14:23:31.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.129+0000 7f04a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:31.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.129+0000 7f04a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:31.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:31.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:31.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:31.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f04bc068950 con 0x7f04bc09bea0 2024-09-18T14:23:31.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04a2ffd700 1 -- 172.21.15.104:0/2207106236 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f04b401b310 con 0x7f04bc09bea0 2024-09-18T14:23:31.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:31.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:31.361 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 >> v2:172.21.15.174:6812/17603 conn(0x7f04a8035a80 msgr2=0x7f04a8037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 --2- 172.21.15.104:0/2207106236 >> v2:172.21.15.174:6812/17603 conn(0x7f04a8035a80 0x7f04a8037f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f04b00030c0 tx=0x7f04b0016000 comp rx=0 tx=0).stop 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 10 monclient: shutdown 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 msgr2=0x7f04bc131e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 0x7f04bc131e20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f04b400e0e0 tx=0x7f04b4009930 comp rx=0 tx=0).stop 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 shutdown_connections 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 --2- 172.21.15.104:0/2207106236 >> v2:172.21.15.174:6812/17603 conn(0x7f04a8035a80 0x7f04a8037f30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04bc0a59d0 0x7f04bc1416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04bc09fcb0 0x7f04bc1411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 --2- 172.21.15.104:0/2207106236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f04bc09bea0 0x7f04bc131e20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:31.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.357+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 >> 172.21.15.104:0/2207106236 conn(0x7f04bc067c10 msgr2=0x7f04bc09d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:31.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.361+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 shutdown_connections 2024-09-18T14:23:31.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:31.361+0000 7f04c08d1700 1 -- 172.21.15.104:0/2207106236 wait complete. 2024-09-18T14:23:31.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19641]: starting. 2024-09-18T14:23:31.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19641]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19640 2024-09-18T14:23:31.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19641]: created object 0... 2024-09-18T14:23:31.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19641]: created object 25... 2024-09-18T14:23:31.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19641]: created object 49... 2024-09-18T14:23:31.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19641]: finishing. 2024-09-18T14:23:31.899 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19641]: shutting down. 2024-09-18T14:23:32.728 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19642]: starting. 2024-09-18T14:23:32.728 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19642]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19640 2024-09-18T14:23:32.728 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19642]: shutting down. 2024-09-18T14:23:34.376 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:34.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.481+0000 7ff97846e700 10 monclient: get_monmap_and_config 2024-09-18T14:23:34.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.481+0000 7ff97846e700 10 monclient: build_initial_monmap 2024-09-18T14:23:34.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.481+0000 7ff97846e700 10 monclient: monmap: 2024-09-18T14:23:34.485 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:34.485130+0000 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:34.485130+0000 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:34.486 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:34.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.481+0000 7ff97846e700 1 Processor -- start 2024-09-18T14:23:34.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.481+0000 7ff97846e700 1 -- start start 2024-09-18T14:23:34.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.481+0000 7ff97846e700 10 monclient: init 2024-09-18T14:23:34.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:34.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T14:23:34.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:34.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient(hunting): picked mon.noname-b con 0x7ff97009f860 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:34.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff97009fc30 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:34.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient(hunting): picked mon.noname-c con 0x7ff970133690 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97006cd00 con 0x7ff97009fc30 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9700a1780 con 0x7ff970133690 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97006ca20 con 0x7ff97009f860 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97846e700 10 monclient(hunting): authenticate will time out at 935.476441s 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:34.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:39146/0 (socket says 172.21.15.104:39146) 2024-09-18T14:23:34.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 learned_addr learned my addr 172.21.15.104:0/380748869 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff97009f860 auth_method 0 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 10 monclient(hunting): get_auth_request con 0x7ff970133690 auth_method 0 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:34.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:34.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:34.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:23:34.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:34.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:34.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:34.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.485+0000 7ff975a09700 1 -- 172.21.15.104:0/380748869 >> v1:172.21.15.104:6789/0 conn(0x7ff97009fc30 legacy=0x7ff9700a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff97009f860 auth_method 0 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 10 monclient(hunting): get_auth_request con 0x7ff970133690 auth_method 0 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:34.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:34.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:34.689+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:23:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff97009f860 auth_method 0 2024-09-18T14:23:35.097 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 10 monclient(hunting): get_auth_request con 0x7ff970133690 auth_method 0 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:35.098 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:23:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:35.099 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.093+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff97009f860 auth_method 0 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 10 monclient(hunting): get_auth_request con 0x7ff970133690 auth_method 0 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:35.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:35.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:35.897+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:23:37.488 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 10 monclient(hunting): tick 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 monclient(hunting): continuing hunt 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 msgr2=0x7ff97006dd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff97009f860 0x7ff97006dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 msgr2=0x7ff970135a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.174:3300/0 conn(0x7ff970133690 0x7ff970135a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 -- 172.21.15.104:0/380748869 >> v1:172.21.15.104:6789/0 conn(0x7ff97009fc30 legacy=0x7ff9700a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 10 monclient: _add_conns ranks=[3,1] 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff954003790 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9540062d0 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:37.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 -- 172.21.15.104:0/380748869 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9700a14e0 con 0x7ff9540062d0 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 1 -- 172.21.15.104:0/380748869 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9700a1780 con 0x7ff954003790 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff967fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff954003790 auth_method 0 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:37.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:37.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.485+0000 7ff975a09700 1 -- 172.21.15.104:0/380748869 >> v1:172.21.15.174:6789/0 conn(0x7ff9540062d0 legacy=0x7ff954008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff954003790 auth_method 0 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:37.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:37.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:37.685+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:23:38.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:38.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff954003790 auth_method 0 2024-09-18T14:23:38.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:38.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:38.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:38.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:38.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:38.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:38.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.085+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff954003790 auth_method 0 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:38.893 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:38.889+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:23:39.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881405684 0 0) 0x7ff9700a14e0 con 0x7ff9540062d0 2024-09-18T14:23:39.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:39.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:39.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): my global_id is 4804 2024-09-18T14:23:39.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff95c003570 con 0x7ff9540062d0 2024-09-18T14:23:39.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2611874311 0 0) 0x7ff95c003570 con 0x7ff9540062d0 2024-09-18T14:23:39.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 msgr2=0x7ff954005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:39.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 --2- 172.21.15.104:0/380748869 >> v2:172.21.15.104:3300/0 conn(0x7ff954003790 0x7ff954005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: _finish_hunting 0 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 monclient: found mon.noname-c-legacy 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.174:6789/0 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95400b010 con 0x7ff9540062d0 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: _finish_auth 0 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7ff960001830 con 0x7ff9540062d0 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97846e700 5 monclient: authenticate success, global_id 4804 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97846e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: dump: 2024-09-18T14:23:39.041 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] but i'm connected to v1:172.21.15.174:6789/0, reconnecting 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 >> v1:172.21.15.174:6789/0 conn(0x7ff9540062d0 legacy=0x7ff954008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff976a0b700 1 -- 172.21.15.104:0/380748869 reap_dead start 2024-09-18T14:23:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:23:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff95c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): picked mon.c con 0x7ff95c003b50 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c004810 0x7ff95c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): picked mon.b con 0x7ff95c004810 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c0053c0 0x7ff95c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): picked mon.a con 0x7ff95c0053c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700a1780 con 0x7ff95c0053c0 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c00c380 con 0x7ff95c004810 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c00c540 con 0x7ff95c003b50 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975208700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97846e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff95c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff95c003b50 auth_method 0 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff975a09700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c004810 0x7ff95c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.033+0000 7ff97620a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c0053c0 0x7ff95c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): get_auth_request con 0x7ff95c004810 auth_method 0 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 10 monclient(hunting): get_auth_request con 0x7ff95c0053c0 auth_method 0 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff97620a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff97620a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff97620a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient(hunting): handle_auth_done global_id 4804 payload 293 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 msgr2=0x7ff95c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff95c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c0053c0 msgr2=0x7ff95c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:39.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c0053c0 0x7ff95c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient: _finish_hunting 0 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 monclient: found mon.b 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 -- 172.21.15.104:0/380748869 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c003570 con 0x7ff95c004810 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 10 monclient: _finish_auth 0 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff97620a700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff95c003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:23:39.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff976a0b700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c0053c0 0x7ff95c00baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:23:39.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c004810 msgr2=0x7ff95c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:39.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.037+0000 7ff975a09700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c004810 0x7ff95c004c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff960001a10 tx=0x7ff960007a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f709f1beff986dbf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:39.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96000cd20 con 0x7ff95c004810 2024-09-18T14:23:39.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:39.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 10 monclient: dump: 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:39.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9600070d0 con 0x7ff95c004810 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 4 set_mon_vals no callback set 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 1 -- 172.21.15.104:0/380748869 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96000cea0 con 0x7ff95c004810 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff975208700 10 monclient: dump: 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:39.053 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 10 monclient: shutdown 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 -- 172.21.15.104:0/380748869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c004810 msgr2=0x7ff95c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c004810 0x7ff95c004c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff960001a10 tx=0x7ff960007a90 comp rx=0 tx=0).stop 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 -- 172.21.15.104:0/380748869 shutdown_connections 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c0053c0 0x7ff95c00baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c004810 0x7ff95c004c80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 --2- 172.21.15.104:0/380748869 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff95c003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 -- 172.21.15.104:0/380748869 >> 172.21.15.104:0/380748869 conn(0x7ff970067ef0 msgr2=0x7ff9700a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:39.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 -- 172.21.15.104:0/380748869 shutdown_connections 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 -- 172.21.15.104:0/380748869 wait complete. 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 10 monclient: build_initial_monmap 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 10 monclient: monmap: 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:39.055 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 Processor -- start 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 1 -- start start 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.041+0000 7ff97846e700 10 monclient: init 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff97013cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): picked mon.noname-c con 0x7ff95c003b50 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 0x7ff97013d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): picked mon.noname-a con 0x7ff95c004810 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c0053c0 0x7ff9701180c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): picked mon.noname-b con 0x7ff95c0053c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97011b530 con 0x7ff95c004810 2024-09-18T14:23:39.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97011a040 con 0x7ff95c0053c0 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97011b730 con 0x7ff95c003b50 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97846e700 10 monclient(hunting): authenticate will time out at 940.037081s 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff97013cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 0x7ff97013d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff97013cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:39202/0 (socket says 172.21.15.104:39202) 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 1 -- 172.21.15.104:0/649630052 learned_addr learned my addr 172.21.15.104:0/649630052 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 10 monclient(hunting): get_auth_request con 0x7ff95c003b50 auth_method 0 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 10 monclient(hunting): get_auth_request con 0x7ff95c004810 auth_method 0 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 msgr2=0x7ff97013cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 msgr2=0x7ff97013cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff97013cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff97620a700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff97013cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 msgr2=0x7ff97013d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 msgr2=0x7ff97013d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c0053c0 0x7ff9701180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 0x7ff97013d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): get_auth_request con 0x7ff95c0053c0 auth_method 0 2024-09-18T14:23:39.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff975a09700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 0x7ff97013d2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient(hunting): handle_auth_done global_id 4807 payload 824 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 msgr2=0x7ff97013cd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff97013cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 msgr2=0x7ff97013d2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 0x7ff97013d2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient: _finish_hunting 0 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 1 monclient: found mon.noname-b 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:39.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:23:39.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.045+0000 7ff976a0b700 1 -- 172.21.15.104:0/649630052 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff970119b50 con 0x7ff95c0053c0 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff976a0b700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c0053c0 msgr2=0x7ff9701180c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff976a0b700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c0053c0 0x7ff9701180c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff9700a0710 tx=0x7ff96800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 1 -- 172.21.15.104:0/649630052 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968018630 con 0x7ff95c0053c0 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: dump: 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.062 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff97846e700 5 monclient: authenticate success, global_id 4807 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 1 -- 172.21.15.104:0/649630052 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96800f040 con 0x7ff95c0053c0 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff97846e700 10 monclient: _renew_subs 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff97846e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97009da70 con 0x7ff95c0053c0 2024-09-18T14:23:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 1 -- 172.21.15.104:0/649630052 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9680189e0 con 0x7ff95c0053c0 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 10 monclient: dump: 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:39.064 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff974a07700 4 set_mon_vals no callback set 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff97846e700 10 monclient: _renew_subs 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff97846e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97011c5c0 con 0x7ff95c0053c0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.049+0000 7ff9667fc700 1 -- 172.21.15.104:0/649630052 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7ff968003040 con 0x7ff95c0053c0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff97846e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff97846e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff97846e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff9667fc700 1 --2- 172.21.15.104:0/649630052 >> v2:172.21.15.174:6812/17603 conn(0x7ff954036430 0x7ff9540388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9700636d0 con 0x7ff95c0053c0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff9667fc700 1 -- 172.21.15.104:0/649630052 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 3894+0+0 (secure 0 0 0) 0x7ff9680494a0 con 0x7ff95c0053c0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff97620a700 1 --2- 172.21.15.104:0/649630052 >> v2:172.21.15.174:6812/17603 conn(0x7ff954036430 0x7ff9540388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff97620a700 10 monclient: get_auth_request con 0x7ff954036430 auth_method 0 2024-09-18T14:23:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.053+0000 7ff97620a700 1 --2- 172.21.15.104:0/649630052 >> v2:172.21.15.174:6812/17603 conn(0x7ff954036430 0x7ff9540388e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff96c006db0 tx=0x7ff96c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:39.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.061+0000 7ff9667fc700 1 -- 172.21.15.104:0/649630052 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff968021460 con 0x7ff95c0053c0 2024-09-18T14:23:39.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.061+0000 7ff9667fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:39.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.061+0000 7ff9667fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:39.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.285+0000 7ff97846e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:39.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:39.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:39.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff97011ade0 con 0x7ff95c0053c0 2024-09-18T14:23:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff9667fc700 1 -- 172.21.15.104:0/649630052 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7ff968029b90 con 0x7ff95c0053c0 2024-09-18T14:23:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff9667fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:39.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff9667fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:39.293 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 >> v2:172.21.15.174:6812/17603 conn(0x7ff954036430 msgr2=0x7ff9540388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 --2- 172.21.15.104:0/649630052 >> v2:172.21.15.174:6812/17603 conn(0x7ff954036430 0x7ff9540388e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff96c006db0 tx=0x7ff96c008040 comp rx=0 tx=0).stop 2024-09-18T14:23:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 10 monclient: shutdown 2024-09-18T14:23:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c0053c0 msgr2=0x7ff9701180c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:39.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c0053c0 0x7ff9701180c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff9700a0710 tx=0x7ff96800b040 comp rx=0 tx=0).stop 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 shutdown_connections 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 --2- 172.21.15.104:0/649630052 >> v2:172.21.15.174:6812/17603 conn(0x7ff954036430 0x7ff9540388e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c0053c0 0x7ff9701180c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff95c004810 0x7ff97013d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 --2- 172.21.15.104:0/649630052 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff95c003b50 0x7ff97013cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 >> 172.21.15.104:0/649630052 conn(0x7ff970067ef0 msgr2=0x7ff97006dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 shutdown_connections 2024-09-18T14:23:39.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:39.289+0000 7ff97846e700 1 -- 172.21.15.104:0/649630052 wait complete. 2024-09-18T14:23:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19677]: starting. 2024-09-18T14:23:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19677]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19640 2024-09-18T14:23:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19677]: created object 0... 2024-09-18T14:23:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19677]: created object 25... 2024-09-18T14:23:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19677]: created object 49... 2024-09-18T14:23:40.807 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19677]: finishing. 2024-09-18T14:23:40.807 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19677]: shutting down. 2024-09-18T14:23:40.867 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:40.867 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19679]: starting. 2024-09-18T14:23:40.867 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19679]: listing objects. 2024-09-18T14:23:40.867 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19679]: listed object 0... 2024-09-18T14:23:40.867 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19679]: listed object 25... 2024-09-18T14:23:40.867 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19679]: saw 50 objects 2024-09-18T14:23:40.867 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19679]: shutting down. 2024-09-18T14:23:41.460 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:41.460 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19678]: starting. 2024-09-18T14:23:41.460 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19678]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19640 2024-09-18T14:23:41.460 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19678]: shutting down. 2024-09-18T14:23:41.461 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:41.461 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:23:41.461 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:23:41.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:23:42.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:42.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.413+0000 7fe4ae468700 10 monclient: get_monmap_and_config 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.413+0000 7fe4ae468700 10 monclient: build_initial_monmap 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.413+0000 7fe4ae468700 10 monclient: monmap: 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:42.417716+0000 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:42.417716+0000 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:42.418 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:42.419 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.413+0000 7fe4ae468700 1 Processor -- start 2024-09-18T14:23:42.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.413+0000 7fe4ae468700 1 -- start start 2024-09-18T14:23:42.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.413+0000 7fe4ae468700 10 monclient: init 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 0x7fe4a809bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient(hunting): picked mon.noname-b con 0x7fe4a809bba0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4a809b0a0 0x7fe4a809ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient(hunting): picked mon.noname-a con 0x7fe4a809b0a0 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4a809b470 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a809e440 con 0x7fe4a809b470 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a8099570 con 0x7fe4a809b0a0 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a809e610 con 0x7fe4a809bba0 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ae468700 10 monclient(hunting): authenticate will time out at 943.409325s 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 0x7fe4a809bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 0x7fe4a809bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59270/0 (socket says 172.21.15.104:59270) 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 -- 172.21.15.104:0/3752516118 learned_addr learned my addr 172.21.15.104:0/3752516118 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): get_auth_request con 0x7fe4a809bba0 auth_method 0 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 1 --2- 172.21.15.104:0/3752516118 >> v2:172.21.15.104:3300/0 conn(0x7fe4a809b0a0 0x7fe4a809ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 10 monclient(hunting): get_auth_request con 0x7fe4a809b0a0 auth_method 0 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4acc65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient(hunting): handle_auth_done global_id 4790 payload 274 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 -- 172.21.15.104:0/3752516118 >> v2:172.21.15.104:3300/0 conn(0x7fe4a809b0a0 msgr2=0x7fe4a809ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 --2- 172.21.15.104:0/3752516118 >> v2:172.21.15.104:3300/0 conn(0x7fe4a809b0a0 0x7fe4a809ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 -- 172.21.15.104:0/3752516118 >> v1:172.21.15.104:6789/0 conn(0x7fe4a809b470 legacy=0x7fe4a809cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient: _finish_hunting 0 2024-09-18T14:23:42.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 monclient: found mon.noname-b 2024-09-18T14:23:42.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:42.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:23:42.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.417+0000 7fe4ad466700 1 -- 172.21.15.104:0/3752516118 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a809a3e0 con 0x7fe4a809bba0 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ad466700 1 -- 172.21.15.104:0/3752516118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 msgr2=0x7fe4a809bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ad466700 1 --2- 172.21.15.104:0/3752516118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 0x7fe4a809bf70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40099b0 tx=0x7fe4a4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96ca9b61c416768c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 1 -- 172.21.15.104:0/3752516118 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4009ea0 con 0x7fe4a809bba0 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: dump: 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.429 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: _finish_auth 0 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 5 monclient: authenticate success, global_id 4790 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 1 -- 172.21.15.104:0/3752516118 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a400dc80 con 0x7fe4a809bba0 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 4 set_mon_vals no callback set 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 1 -- 172.21.15.104:0/3752516118 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4010470 con 0x7fe4a809bba0 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe49ffff700 10 monclient: dump: 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient: shutdown 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- 172.21.15.104:0/3752516118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 msgr2=0x7fe4a809bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 --2- 172.21.15.104:0/3752516118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 0x7fe4a809bf70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40099b0 tx=0x7fe4a4009250 comp rx=0 tx=0).stop 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- 172.21.15.104:0/3752516118 shutdown_connections 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 --2- 172.21.15.104:0/3752516118 >> v2:172.21.15.104:3300/0 conn(0x7fe4a809b0a0 0x7fe4a809ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 --2- 172.21.15.104:0/3752516118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809bba0 0x7fe4a809bf70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- 172.21.15.104:0/3752516118 >> 172.21.15.104:0/3752516118 conn(0x7fe4a8068e60 msgr2=0x7fe4a80a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- 172.21.15.104:0/3752516118 shutdown_connections 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- 172.21.15.104:0/3752516118 wait complete. 2024-09-18T14:23:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient: build_initial_monmap 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient: monmap: 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 Processor -- start 2024-09-18T14:23:42.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- start start 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient: init 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a809b0a0 0x7fe4a806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): picked mon.noname-a con 0x7fe4a809b0a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809b470 0x7fe4a812da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): picked mon.noname-c con 0x7fe4a809b470 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a809bba0 0x7fe4a81186f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): picked mon.noname-b con 0x7fe4a809bba0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a806eb40 con 0x7fe4a809b0a0 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a811ae70 con 0x7fe4a809bba0 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a811ab90 con 0x7fe4a809b470 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ae468700 10 monclient(hunting): authenticate will time out at 943.415734s 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4acc65700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809b470 0x7fe4a812da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4acc65700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809b470 0x7fe4a812da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59286/0 (socket says 172.21.15.104:59286) 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4acc65700 1 -- 172.21.15.104:0/1873109714 learned_addr learned my addr 172.21.15.104:0/1873109714 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4acc65700 10 monclient(hunting): get_auth_request con 0x7fe4a809b470 auth_method 0 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4ad466700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a809b0a0 0x7fe4a806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4acc65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4acc65700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.421+0000 7fe4acc65700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): get_auth_request con 0x7fe4a809b0a0 auth_method 0 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4acc65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4acc65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4adc67700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a809bba0 0x7fe4a81186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4acc65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4adc67700 10 monclient(hunting): get_auth_request con 0x7fe4a809bba0 auth_method 0 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4adc67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4adc67700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4adc67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient(hunting): handle_auth_done global_id 14115 payload 824 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 -- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809b470 msgr2=0x7fe4a812da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809b470 0x7fe4a812da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 -- 172.21.15.104:0/1873109714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a809bba0 msgr2=0x7fe4a81186f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a809bba0 0x7fe4a81186f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient: _finish_hunting 0 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 monclient: found mon.noname-a 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 -- 172.21.15.104:0/1873109714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40094f0 con 0x7fe4a809b0a0 2024-09-18T14:23:42.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4adc67700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a809bba0 0x7fe4a81186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:23:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 -- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a809b0a0 msgr2=0x7fe4a806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ad466700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a809b0a0 0x7fe4a806dfe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4a809a750 tx=0x7fe4a4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 1 -- 172.21.15.104:0/1873109714 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4018420 con 0x7fe4a809b0a0 2024-09-18T14:23:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:42.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: dump: 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: _finish_auth 0 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 1 -- 172.21.15.104:0/1873109714 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a401e040 con 0x7fe4a809b0a0 2024-09-18T14:23:42.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ae468700 5 monclient: authenticate success, global_id 14115 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ae468700 10 monclient: _renew_subs 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ae468700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a811b150 con 0x7fe4a809b0a0 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 1 -- 172.21.15.104:0/1873109714 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4018810 con 0x7fe4a809b0a0 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49d7fa700 10 monclient: dump: 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.441 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ae468700 10 monclient: _renew_subs 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe49f7fe700 4 set_mon_vals no callback set 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ae468700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.425+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a813dcc0 con 0x7fe4a809b0a0 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe4ae468700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe4ae468700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe4ae468700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a811acd0 con 0x7fe4a809b0a0 2024-09-18T14:23:42.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe49d7fa700 1 -- 172.21.15.104:0/1873109714 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fe4a4018c00 con 0x7fe4a809b0a0 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe49d7fa700 1 --2- 172.21.15.104:0/1873109714 >> v2:172.21.15.174:6812/17603 conn(0x7fe48c035ad0 0x7fe48c037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe49d7fa700 1 -- 172.21.15.104:0/1873109714 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe4a4014070 con 0x7fe4a809b0a0 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe4acc65700 1 --2- 172.21.15.104:0/1873109714 >> v2:172.21.15.174:6812/17603 conn(0x7fe48c035ad0 0x7fe48c037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.429+0000 7fe4acc65700 10 monclient: get_auth_request con 0x7fe48c035ad0 auth_method 0 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.433+0000 7fe4acc65700 1 --2- 172.21.15.104:0/1873109714 >> v2:172.21.15.174:6812/17603 conn(0x7fe48c035ad0 0x7fe48c037f80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8063f50 tx=0x7fe4980091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.441+0000 7fe49d7fa700 1 -- 172.21.15.104:0/1873109714 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe4a811acd0 con 0x7fe4a809b0a0 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.441+0000 7fe49d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:42.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.441+0000 7fe49d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:42.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.665+0000 7fe4ae468700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:42.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.665+0000 7fe4ae468700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:42.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.665+0000 7fe4ae468700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.665+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe4a811acd0 con 0x7fe4a809b0a0 2024-09-18T14:23:42.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe49d7fa700 1 -- 172.21.15.104:0/1873109714 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fe4a811acd0 con 0x7fe4a809b0a0 2024-09-18T14:23:42.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe49d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:42.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe49d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:42.673 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 >> v2:172.21.15.174:6812/17603 conn(0x7fe48c035ad0 msgr2=0x7fe48c037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 --2- 172.21.15.104:0/1873109714 >> v2:172.21.15.174:6812/17603 conn(0x7fe48c035ad0 0x7fe48c037f80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8063f50 tx=0x7fe4980091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 10 monclient: shutdown 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a809b0a0 msgr2=0x7fe4a806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a809b0a0 0x7fe4a806dfe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4a809a750 tx=0x7fe4a4018000 comp rx=0 tx=0).stop 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 shutdown_connections 2024-09-18T14:23:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 --2- 172.21.15.104:0/1873109714 >> v2:172.21.15.174:6812/17603 conn(0x7fe48c035ad0 0x7fe48c037f80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a809bba0 0x7fe4a81186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe4a809b470 0x7fe4a812da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 --2- 172.21.15.104:0/1873109714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a809b0a0 0x7fe4a806dfe0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 >> 172.21.15.104:0/1873109714 conn(0x7fe4a8068e60 msgr2=0x7fe4a809f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 shutdown_connections 2024-09-18T14:23:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.669+0000 7fe4ae468700 1 -- 172.21.15.104:0/1873109714 wait complete. 2024-09-18T14:23:42.686 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T14:23:42.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T14:23:42.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:23:42.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.785+0000 7fa1ef21d700 10 monclient: get_monmap_and_config 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.785+0000 7fa1ef21d700 10 monclient: build_initial_monmap 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.785+0000 7fa1ef21d700 10 monclient: monmap: 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:42.790777+0000 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:42.790777+0000 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 1 Processor -- start 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 1 -- start start 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient: init 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T14:23:42.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 0x7fa1e806f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient(hunting): picked mon.noname-b con 0x7fa1e809a4d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa1e806f580 0x7fa1e809fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient(hunting): picked mon.noname-c con 0x7fa1e806f580 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1e80a02e0 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e809b490 con 0x7fa1e80a02e0 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e809b850 con 0x7fa1e806f580 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e809a8a0 con 0x7fa1e809a4d0 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ef21d700 10 monclient(hunting): authenticate will time out at 943.781151s 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 0x7fa1e806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 0x7fa1e806f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59288/0 (socket says 172.21.15.104:59288) 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 -- 172.21.15.104:0/2498927427 learned_addr learned my addr 172.21.15.104:0/2498927427 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): get_auth_request con 0x7fa1e809a4d0 auth_method 0 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eea1c700 1 --1- 172.21.15.104:0/2498927427 >> v1:172.21.15.104:6789/0 conn(0x7fa1e80a02e0 0x7fa1e80a0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59764/0 (socket says 172.21.15.104:59764) 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 1 --2- 172.21.15.104:0/2498927427 >> v2:172.21.15.174:3300/0 conn(0x7fa1e806f580 0x7fa1e809fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 10 monclient(hunting): get_auth_request con 0x7fa1e806f580 auth_method 0 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1eda1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_done global_id 4796 payload 274 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 -- 172.21.15.104:0/2498927427 >> v2:172.21.15.174:3300/0 conn(0x7fa1e806f580 msgr2=0x7fa1e809fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 --2- 172.21.15.104:0/2498927427 >> v2:172.21.15.174:3300/0 conn(0x7fa1e806f580 0x7fa1e809fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 -- 172.21.15.104:0/2498927427 >> v1:172.21.15.104:6789/0 conn(0x7fa1e80a02e0 legacy=0x7fa1e80a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:42.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient: _finish_hunting 0 2024-09-18T14:23:42.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 monclient: found mon.noname-b 2024-09-18T14:23:42.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:42.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:23:42.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.789+0000 7fa1ee21b700 1 -- 172.21.15.104:0/2498927427 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e809b1b0 con 0x7fa1e809a4d0 2024-09-18T14:23:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ee21b700 1 -- 172.21.15.104:0/2498927427 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 msgr2=0x7fa1e806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ee21b700 1 --2- 172.21.15.104:0/2498927427 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 0x7fa1e806f040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40099b0 tx=0x7fa1e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dbf534dc0efc12f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 1 -- 172.21.15.104:0/2498927427 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4009ea0 con 0x7fa1e809a4d0 2024-09-18T14:23:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:23:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: dump: 2024-09-18T14:23:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: _finish_auth 0 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 1 -- 172.21.15.104:0/2498927427 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e400ba70 con 0x7fa1e809a4d0 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 5 monclient: authenticate success, global_id 4796 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 4 set_mon_vals no callback set 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 1 -- 172.21.15.104:0/2498927427 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400be10 con 0x7fa1e809a4d0 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ed219700 10 monclient: dump: 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 10 monclient: shutdown 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 -- 172.21.15.104:0/2498927427 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 msgr2=0x7fa1e806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 --2- 172.21.15.104:0/2498927427 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 0x7fa1e806f040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40099b0 tx=0x7fa1e4009250 comp rx=0 tx=0).stop 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 -- 172.21.15.104:0/2498927427 shutdown_connections 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 --2- 172.21.15.104:0/2498927427 >> v2:172.21.15.174:3300/0 conn(0x7fa1e806f580 0x7fa1e809fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 --2- 172.21.15.104:0/2498927427 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e809a4d0 0x7fa1e806f040 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 -- 172.21.15.104:0/2498927427 >> 172.21.15.104:0/2498927427 conn(0x7fa1e80a6010 msgr2=0x7fa1e80676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 -- 172.21.15.104:0/2498927427 shutdown_connections 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 -- 172.21.15.104:0/2498927427 wait complete. 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 10 monclient: build_initial_monmap 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 10 monclient: monmap: 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:42.811 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.793+0000 7fa1ef21d700 1 Processor -- start 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 1 -- start start 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient: init 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:42.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e806f580 0x7fa1e8141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): picked mon.noname-b con 0x7fa1e806f580 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e809a4d0 0x7fa1e8133e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): picked mon.noname-a con 0x7fa1e809a4d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e80a02e0 0x7fa1e811d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): picked mon.noname-c con 0x7fa1e80a02e0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8120f40 con 0x7fa1e809a4d0 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e806f160 con 0x7fa1e806f580 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e809ace0 con 0x7fa1e80a02e0 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ef21d700 10 monclient(hunting): authenticate will time out at 943.788955s 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e809a4d0 0x7fa1e8133e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e809a4d0 0x7fa1e8133e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53482/0 (socket says 172.21.15.104:53482) 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 -- 172.21.15.104:0/3832789319 learned_addr learned my addr 172.21.15.104:0/3832789319 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): get_auth_request con 0x7fa1e809a4d0 auth_method 0 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e80a02e0 0x7fa1e811d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e806f580 0x7fa1e8141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 10 monclient(hunting): get_auth_request con 0x7fa1e80a02e0 auth_method 0 2024-09-18T14:23:42.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 10 monclient(hunting): get_auth_request con 0x7fa1e806f580 auth_method 0 2024-09-18T14:23:42.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eea1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient(hunting): handle_auth_done global_id 14118 payload 824 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e80a02e0 msgr2=0x7fa1e811d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1e80a02e0 0x7fa1e811d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e806f580 msgr2=0x7fa1e8141140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1e806f580 0x7fa1e8141140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient: _finish_hunting 0 2024-09-18T14:23:42.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 monclient: found mon.noname-a 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.797+0000 7fa1eda1a700 1 -- 172.21.15.104:0/3832789319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40094f0 con 0x7fa1e809a4d0 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1eda1a700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e809a4d0 msgr2=0x7fa1e8133e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1eda1a700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e809a4d0 0x7fa1e8133e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800ecb0 tx=0x7fa1d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800ca00 con 0x7fa1e809a4d0 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: dump: 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.819 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d800cb60 con 0x7fa1e809a4d0 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 5 monclient: authenticate success, global_id 14118 2024-09-18T14:23:42.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8013470 con 0x7fa1e809a4d0 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 10 monclient: _renew_subs 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e811f920 con 0x7fa1e809a4d0 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1eca18700 4 set_mon_vals no callback set 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 10 monclient: dump: 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.821 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 10 monclient: _renew_subs 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e811f410 con 0x7fa1e809a4d0 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 0x7fa1e811dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8064830 con 0x7fa1e8134360 2024-09-18T14:23:42.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 0x7fa1e811dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient: get_auth_request con 0x7fa1e8134360 auth_method 0 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fa1d80138a0 con 0x7fa1e809a4d0 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 1 --2- 172.21.15.104:0/3832789319 >> v2:172.21.15.174:6812/17603 conn(0x7fa1cc035a80 0x7fa1cc037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fa1d80482c0 con 0x7fa1e809a4d0 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_done global_id 14121 payload 824 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1ee21b700 1 -- 172.21.15.104:0/3832789319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1e8063450 con 0x7fa1e8134360 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1eea1c700 1 --2- 172.21.15.104:0/3832789319 >> v2:172.21.15.174:6812/17603 conn(0x7fa1cc035a80 0x7fa1cc037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.801+0000 7fa1eea1c700 10 monclient: get_auth_request con 0x7fa1cc035a80 auth_method 0 2024-09-18T14:23:42.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1ee21b700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 msgr2=0x7fa1e811dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1ee21b700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 0x7fa1e811dea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4015000 tx=0x7fa1e4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4021070 con 0x7fa1e8134360 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 10 monclient: dump: 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1eea1c700 1 --2- 172.21.15.104:0/3832789319 >> v2:172.21.15.174:6812/17603 conn(0x7fa1cc035a80 0x7fa1cc037f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa1e811ff70 tx=0x7fa1e0005340 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa1e4026470 con 0x7fa1e8134360 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 msgr2=0x7fa1e811dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 0x7fa1e811dea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4015000 tx=0x7fa1e4016000 comp rx=0 tx=0).stop 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 msgr2=0x7fa1e811dea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.805+0000 7fa1de7fc700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e8134360 0x7fa1e811dea0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4015000 tx=0x7fa1e4016000 comp rx=0 tx=0).stop 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.813+0000 7fa1ef21d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.813+0000 7fa1ef21d700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 0x7fa1e806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.813+0000 7fa1ef21d700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.813+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8134730 con 0x7fa1e811e3e0 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.813+0000 7fa1ee21b700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 0x7fa1e806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.813+0000 7fa1ee21b700 10 monclient: get_auth_request con 0x7fa1e811e3e0 auth_method 0 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.813+0000 7fa1ee21b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 10 monclient(hunting): handle_auth_done global_id 14124 payload 824 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 1 -- 172.21.15.104:0/3832789319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa1e811e7e0 con 0x7fa1e811e3e0 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 msgr2=0x7fa1e806ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1ee21b700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 0x7fa1e806ce00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009290 tx=0x7fa1e401e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e401cdc0 con 0x7fa1e811e3e0 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 10 monclient: dump: 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fa1e4033480 con 0x7fa1e811e3e0 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:23:42.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 msgr2=0x7fa1e806ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.817+0000 7fa1de7fc700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 0x7fa1e806ce00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009290 tx=0x7fa1e401e070 comp rx=0 tx=0).stop 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1de7fc700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 msgr2=0x7fa1e806ce00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1de7fc700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e811e3e0 0x7fa1e806ce00 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009290 tx=0x7fa1e401e070 comp rx=0 tx=0).stop 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 20, 2024-09-18T14:23:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 2, 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:23:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:23:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:23:42.843 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:23:42.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 >> v2:172.21.15.174:6812/17603 conn(0x7fa1cc035a80 msgr2=0x7fa1cc037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 --2- 172.21.15.104:0/3832789319 >> v2:172.21.15.174:6812/17603 conn(0x7fa1cc035a80 0x7fa1cc037f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa1e811ff70 tx=0x7fa1e0005340 comp rx=0 tx=0).stop 2024-09-18T14:23:42.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 10 monclient: shutdown 2024-09-18T14:23:42.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:42.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e809a4d0 msgr2=0x7fa1e8133e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 --2- 172.21.15.104:0/3832789319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e809a4d0 0x7fa1e8133e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800ecb0 tx=0x7fa1d80091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1eea1c700 1 -- 172.21.15.104:0/3832789319 reap_dead start 2024-09-18T14:23:42.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1eea1c700 1 -- 172.21.15.104:0/3832789319 reap_dead start 2024-09-18T14:23:42.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 shutdown_connections 2024-09-18T14:23:42.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 >> 172.21.15.104:0/3832789319 conn(0x7fa1e80a6010 msgr2=0x7fa1e8134c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:42.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 shutdown_connections 2024-09-18T14:23:42.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.821+0000 7fa1ef21d700 1 -- 172.21.15.104:0/3832789319 wait complete. 2024-09-18T14:23:42.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:23:42.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T14:23:42.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T14:23:42.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient: get_monmap_and_config 2024-09-18T14:23:42.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient: build_initial_monmap 2024-09-18T14:23:42.960 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient: monmap: 2024-09-18T14:23:42.960 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:42.960 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:42.960288+0000 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:42.960288+0000 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:42.961 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 1 Processor -- start 2024-09-18T14:23:42.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 1 -- start start 2024-09-18T14:23:42.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient: init 2024-09-18T14:23:42.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:42.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T14:23:42.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f106409e830 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:42.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f106409ec00 0x7f106409f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient(hunting): picked mon.noname-c con 0x7f106409ec00 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1064136e60 0x7f1064139240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient(hunting): picked mon.noname-a con 0x7f1064136e60 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106409b7a0 con 0x7f106409e830 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1064099f10 con 0x7f1064136e60 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106409b4c0 con 0x7f106409ec00 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1063fff700 10 monclient(hunting): authenticate will time out at 943.950932s 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1064136e60 0x7f1064139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1062ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f106409e830 0x7f1064131d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59776/0 (socket says 172.21.15.104:59776) 2024-09-18T14:23:42.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f1062ffd700 1 -- 172.21.15.104:0/830692759 learned_addr learned my addr 172.21.15.104:0/830692759 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): get_auth_request con 0x7f1064136e60 auth_method 0 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 1 --2- 172.21.15.104:0/830692759 >> v2:172.21.15.174:3300/0 conn(0x7f106409ec00 0x7f106409f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 10 monclient(hunting): get_auth_request con 0x7f106409ec00 auth_method 0 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient(hunting): handle_auth_done global_id 14127 payload 274 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 1 -- 172.21.15.104:0/830692759 >> v2:172.21.15.174:3300/0 conn(0x7f106409ec00 msgr2=0x7f106409f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 1 --2- 172.21.15.104:0/830692759 >> v2:172.21.15.174:3300/0 conn(0x7f106409ec00 0x7f106409f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 1 -- 172.21.15.104:0/830692759 >> v1:172.21.15.104:6789/0 conn(0x7f106409e830 legacy=0x7f1064131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient: _finish_hunting 0 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 1 monclient: found mon.noname-a 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:42.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.957+0000 7f10637fe700 1 -- 172.21.15.104:0/830692759 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106409ba80 con 0x7f1064136e60 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f10637fe700 1 -- 172.21.15.104:0/830692759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1064136e60 msgr2=0x7f1064139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f10637fe700 1 --2- 172.21.15.104:0/830692759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1064136e60 0x7f1064139240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f106409c1b0 tx=0x7f105c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6405b794d699bd44 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 1 -- 172.21.15.104:0/830692759 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00e070 con 0x7f1064136e60 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: dump: 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.968 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: _finish_auth 0 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 1 -- 172.21.15.104:0/830692759 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c00bd30 con 0x7f1064136e60 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 5 monclient: authenticate success, global_id 14127 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:42.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 4 set_mon_vals no callback set 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 1 -- 172.21.15.104:0/830692759 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00bea0 con 0x7f1064136e60 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1061ffb700 10 monclient: dump: 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.970 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 10 monclient: shutdown 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 -- 172.21.15.104:0/830692759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1064136e60 msgr2=0x7f1064139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 --2- 172.21.15.104:0/830692759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1064136e60 0x7f1064139240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f106409c1b0 tx=0x7f105c009780 comp rx=0 tx=0).stop 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 -- 172.21.15.104:0/830692759 shutdown_connections 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 --2- 172.21.15.104:0/830692759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1064136e60 0x7f1064139240 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 --2- 172.21.15.104:0/830692759 >> v2:172.21.15.174:3300/0 conn(0x7f106409ec00 0x7f106409f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 -- 172.21.15.104:0/830692759 >> 172.21.15.104:0/830692759 conn(0x7f10640676a0 msgr2=0x7f1064067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 -- 172.21.15.104:0/830692759 shutdown_connections 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.961+0000 7f1063fff700 1 -- 172.21.15.104:0/830692759 wait complete. 2024-09-18T14:23:42.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient: build_initial_monmap 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient: monmap: 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 Processor -- start 2024-09-18T14:23:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 -- start start 2024-09-18T14:23:42.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient: init 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106409e830 0x7f1064136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): picked mon.noname-c con 0x7f106409e830 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f106409ec00 0x7f106412f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): picked mon.noname-a con 0x7f106409ec00 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1064136e60 0x7f106412f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): picked mon.noname-b con 0x7f1064136e60 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106409f5e0 con 0x7f106409ec00 2024-09-18T14:23:42.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1064130b90 con 0x7f1064136e60 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10641312e0 con 0x7f106409e830 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1063fff700 10 monclient(hunting): authenticate will time out at 943.957507s 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106409e830 0x7f1064136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f106409ec00 0x7f106412f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106409e830 0x7f1064136380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59314/0 (socket says 172.21.15.104:59314) 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 -- 172.21.15.104:0/640090666 learned_addr learned my addr 172.21.15.104:0/640090666 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): get_auth_request con 0x7f106409e830 auth_method 0 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 10 monclient(hunting): get_auth_request con 0x7f106409ec00 auth_method 0 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1064136e60 0x7f106412f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 10 monclient(hunting): get_auth_request con 0x7f1064136e60 auth_method 0 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient(hunting): handle_auth_done global_id 4802 payload 824 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1064136e60 msgr2=0x7f106412f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1064136e60 0x7f106412f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f106409ec00 msgr2=0x7f106412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f106409ec00 0x7f106412f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient: _finish_hunting 0 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 monclient: found mon.noname-c 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:42.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f1062ffd700 1 -- 172.21.15.104:0/640090666 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c009a20 con 0x7f106409e830 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.965+0000 7f10627fc700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f106409ec00 0x7f106412f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1062ffd700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106409e830 msgr2=0x7f1064136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1062ffd700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106409e830 0x7f1064136380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f105400c560 tx=0x7f1054004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054005d40 con 0x7f106409e830 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: dump: 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: _finish_auth 0 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 5 monclient: authenticate success, global_id 4802 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10540070d0 con 0x7f106409e830 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054007490 con 0x7f106409e830 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 10 monclient: _renew_subs 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:42.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1064133430 con 0x7f106409e830 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 10 monclient: dump: 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10617fa700 4 set_mon_vals no callback set 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 10 monclient: _renew_subs 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1064131760 con 0x7f106409e830 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 0x7f10640a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1064064840 con 0x7f106412fda0 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f105401b3b0 con 0x7f106409e830 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 0x7f10640a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient: get_auth_request con 0x7f106412fda0 auth_method 0 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 1 --2- 172.21.15.104:0/640090666 >> v2:172.21.15.174:6812/17603 conn(0x7f1050035a80 0x7f1050037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f1054047480 con 0x7f106409e830 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 10 monclient(hunting): handle_auth_done global_id 4805 payload 824 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10627fc700 1 -- 172.21.15.104:0/640090666 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1064130170 con 0x7f106412fda0 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10637fe700 1 --2- 172.21.15.104:0/640090666 >> v2:172.21.15.174:6812/17603 conn(0x7f1050035a80 0x7f1050037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.969+0000 7f10637fe700 10 monclient: get_auth_request con 0x7f1050035a80 auth_method 0 2024-09-18T14:23:42.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f10627fc700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 msgr2=0x7f10640a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f10627fc700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 0x7f10640a3e40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1058009f90 tx=0x7f10580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800fec0 con 0x7f106412fda0 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 10 monclient: dump: 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.986 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f10637fe700 1 --2- 172.21.15.104:0/640090666 >> v2:172.21.15.174:6812/17603 conn(0x7f1050035a80 0x7f1050037f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f106409c1b0 tx=0x7f105c019000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1058014860 con 0x7f106412fda0 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 msgr2=0x7f10640a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 0x7f10640a3e40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1058009f90 tx=0x7f10580091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 msgr2=0x7f10640a3e40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:42.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.973+0000 7f104affd700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106412fda0 0x7f10640a3e40 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1058009f90 tx=0x7f10580091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f1063fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:23:42.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f1063fff700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 0x7f10640a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:42.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f1063fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:42.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1064063450 con 0x7f10640a4380 2024-09-18T14:23:42.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 0x7f10640a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:42.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient: get_auth_request con 0x7f10640a4380 auth_method 0 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 10 monclient(hunting): handle_auth_done global_id 4808 payload 824 2024-09-18T14:23:42.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.985+0000 7f10627fc700 1 -- 172.21.15.104:0/640090666 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10640689c0 con 0x7f10640a4380 2024-09-18T14:23:42.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f10627fc700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 msgr2=0x7f10640a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:42.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f10627fc700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 0x7f10640a4780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1058009f90 tx=0x7f105801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:42.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105801c5f0 con 0x7f10640a4380 2024-09-18T14:23:42.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:42.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:42.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 10 monclient: dump: 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:42.993 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:42.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f105801c750 con 0x7f10640a4380 2024-09-18T14:23:42.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:23:42.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:42.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 msgr2=0x7f10640a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:42.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 0x7f10640a4780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1058009f90 tx=0x7f105801c000 comp rx=0 tx=0).stop 2024-09-18T14:23:42.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 msgr2=0x7f10640a4780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:42.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f104affd700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10640a4380 0x7f10640a4780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 20, 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 2, 2024-09-18T14:23:42.995 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:42.996 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:42.997 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:23:42.998 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:42.999 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:23:43.000 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:43.001 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:43.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.005 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.006 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.007 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:23:43.008 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:23:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 >> v2:172.21.15.174:6812/17603 conn(0x7f1050035a80 msgr2=0x7f1050037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 --2- 172.21.15.104:0/640090666 >> v2:172.21.15.174:6812/17603 conn(0x7f1050035a80 0x7f1050037f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f106409c1b0 tx=0x7f105c019000 comp rx=0 tx=0).stop 2024-09-18T14:23:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 10 monclient: shutdown 2024-09-18T14:23:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106409e830 msgr2=0x7f1064136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 --2- 172.21.15.104:0/640090666 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f106409e830 0x7f1064136380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f105400c560 tx=0x7f1054004a80 comp rx=0 tx=0).stop 2024-09-18T14:23:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f10637fe700 1 -- 172.21.15.104:0/640090666 reap_dead start 2024-09-18T14:23:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f10637fe700 1 -- 172.21.15.104:0/640090666 reap_dead start 2024-09-18T14:23:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 shutdown_connections 2024-09-18T14:23:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 >> 172.21.15.104:0/640090666 conn(0x7f10640676a0 msgr2=0x7f106409d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 shutdown_connections 2024-09-18T14:23:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:42.989+0000 7f1063fff700 1 -- 172.21.15.104:0/640090666 wait complete. 2024-09-18T14:23:43.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:23:43.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T14:23:43.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:23:43.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient: get_monmap_and_config 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient: build_initial_monmap 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient: monmap: 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:43.119750+0000 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:43.119750+0000 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:43.120 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:43.121 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 1 Processor -- start 2024-09-18T14:23:43.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 1 -- start start 2024-09-18T14:23:43.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient: init 2024-09-18T14:23:43.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf3409d7f0 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf3409dbc0 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbf341351b0 0x7fbf34137590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient(hunting): picked mon.noname-c con 0x7fbf341351b0 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3409e9f0 con 0x7fbf3409d7f0 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3409aa70 con 0x7fbf3409dbc0 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3409b030 con 0x7fbf341351b0 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf38996700 10 monclient(hunting): authenticate will time out at 944.110094s 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf3259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbf3409d7f0 0x7fbf3409a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59792/0 (socket says 172.21.15.104:59792) 2024-09-18T14:23:43.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf3259c700 1 -- 172.21.15.104:0/585568106 learned_addr learned my addr 172.21.15.104:0/585568106 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 1 --2- 172.21.15.104:0/585568106 >> v2:172.21.15.174:3300/0 conn(0x7fbf341351b0 0x7fbf34137590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): get_auth_request con 0x7fbf341351b0 auth_method 0 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_done global_id 4828 payload 274 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 1 -- 172.21.15.104:0/585568106 >> v1:172.21.15.174:6789/0 conn(0x7fbf3409dbc0 legacy=0x7fbf34133a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 1 -- 172.21.15.104:0/585568106 >> v1:172.21.15.104:6789/0 conn(0x7fbf3409d7f0 legacy=0x7fbf3409a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:23:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 1 monclient: found mon.noname-c 2024-09-18T14:23:43.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:43.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:23:43.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.117+0000 7fbf32d9d700 1 -- 172.21.15.104:0/585568106 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3409dfc0 con 0x7fbf341351b0 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf32d9d700 1 -- 172.21.15.104:0/585568106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf341351b0 msgr2=0x7fbf34137590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf32d9d700 1 --2- 172.21.15.104:0/585568106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf341351b0 0x7fbf34137590 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098e0 tx=0x7fbf2c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e330ddb5af2948 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 1 -- 172.21.15.104:0/585568106 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c00ba70 con 0x7fbf341351b0 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: dump: 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: _finish_auth 0 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 5 monclient: authenticate success, global_id 4828 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 1 -- 172.21.15.104:0/585568106 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c00bbd0 con 0x7fbf341351b0 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:43.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 1 -- 172.21.15.104:0/585568106 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c011420 con 0x7fbf341351b0 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 4 set_mon_vals no callback set 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf3159a700 10 monclient: dump: 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.128 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 10 monclient: shutdown 2024-09-18T14:23:43.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 1 -- 172.21.15.104:0/585568106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf341351b0 msgr2=0x7fbf34137590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 1 --2- 172.21.15.104:0/585568106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf341351b0 0x7fbf34137590 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098e0 tx=0x7fbf2c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 1 -- 172.21.15.104:0/585568106 shutdown_connections 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 1 --2- 172.21.15.104:0/585568106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf341351b0 0x7fbf34137590 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 1 -- 172.21.15.104:0/585568106 >> 172.21.15.104:0/585568106 conn(0x7fbf34068f90 msgr2=0x7fbf340a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 1 -- 172.21.15.104:0/585568106 shutdown_connections 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.121+0000 7fbf38996700 1 -- 172.21.15.104:0/585568106 wait complete. 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient: build_initial_monmap 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient: monmap: 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 Processor -- start 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 -- start start 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient: init 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3409d7f0 0x7fbf3413bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): picked mon.noname-b con 0x7fbf3409d7f0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf3409dbc0 0x7fbf34134710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): picked mon.noname-c con 0x7fbf3409dbc0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf341351b0 0x7fbf3413f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): picked mon.noname-a con 0x7fbf341351b0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3413c7c0 con 0x7fbf341351b0 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3406f160 con 0x7fbf3409d7f0 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3409ef80 con 0x7fbf3409dbc0 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf38996700 10 monclient(hunting): authenticate will time out at 944.118091s 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf341351b0 0x7fbf3413f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf341351b0 0x7fbf3413f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53510/0 (socket says 172.21.15.104:53510) 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 -- 172.21.15.104:0/144279843 learned_addr learned my addr 172.21.15.104:0/144279843 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): get_auth_request con 0x7fbf341351b0 auth_method 0 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf3409dbc0 0x7fbf34134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3409d7f0 0x7fbf3413bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 10 monclient(hunting): get_auth_request con 0x7fbf3409dbc0 auth_method 0 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 10 monclient(hunting): get_auth_request con 0x7fbf3409d7f0 auth_method 0 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf31d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient(hunting): handle_auth_done global_id 14133 payload 824 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf3409dbc0 msgr2=0x7fbf34134710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf3409dbc0 0x7fbf34134710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3409d7f0 msgr2=0x7fbf3413bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3409d7f0 0x7fbf3413bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:23:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 monclient: found mon.noname-a 2024-09-18T14:23:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.125+0000 7fbf32d9d700 1 -- 172.21.15.104:0/144279843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c009420 con 0x7fbf341351b0 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf32d9d700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf341351b0 msgr2=0x7fbf3413f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf32d9d700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf341351b0 0x7fbf3413f120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0093f0 tx=0x7fbf2c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c01b450 con 0x7fbf341351b0 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: dump: 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c020070 con 0x7fbf341351b0 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 5 monclient: authenticate success, global_id 14133 2024-09-18T14:23:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 10 monclient: _renew_subs 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf34119a90 con 0x7fbf341351b0 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c01b7c0 con 0x7fbf341351b0 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 10 monclient: dump: 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf30d99700 4 set_mon_vals no callback set 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 10 monclient: _renew_subs 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf34118900 con 0x7fbf341351b0 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fbf2c022030 con 0x7fbf341351b0 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 0x7fbf3413f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf34119950 con 0x7fbf34138000 2024-09-18T14:23:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.129+0000 7fbf2a7fc700 1 --2- 172.21.15.104:0/144279843 >> v2:172.21.15.174:6812/17603 conn(0x7fbf14035a80 0x7fbf14037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fbf2c049f70 con 0x7fbf341351b0 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 0x7fbf3413f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient: get_auth_request con 0x7fbf34138000 auth_method 0 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf31d9b700 1 --2- 172.21.15.104:0/144279843 >> v2:172.21.15.174:6812/17603 conn(0x7fbf14035a80 0x7fbf14037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf31d9b700 10 monclient: get_auth_request con 0x7fbf14035a80 auth_method 0 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 10 monclient(hunting): handle_auth_done global_id 4834 payload 824 2024-09-18T14:23:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 1 -- 172.21.15.104:0/144279843 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf34118af0 con 0x7fbf34138000 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf31d9b700 1 --2- 172.21.15.104:0/144279843 >> v2:172.21.15.174:6812/17603 conn(0x7fbf14035a80 0x7fbf14037f30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbf341386a0 tx=0x7fbf1c007ea0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 msgr2=0x7fbf3413f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf3259c700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 0x7fbf3413f660 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400e320 tx=0x7fbf2400d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24010040 con 0x7fbf34138000 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf2a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.133+0000 7fbf2a7fc700 10 monclient: dump: 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.145 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.137+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbf2400ee40 con 0x7fbf34138000 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.137+0000 7fbf2a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.137+0000 7fbf2a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.137+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 msgr2=0x7fbf3413f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.137+0000 7fbf2a7fc700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 0x7fbf3413f660 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400e320 tx=0x7fbf2400d040 comp rx=0 tx=0).stop 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.137+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 msgr2=0x7fbf3413f660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.137+0000 7fbf2a7fc700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf34138000 0x7fbf3413f660 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400e320 tx=0x7fbf2400d040 comp rx=0 tx=0).stop 2024-09-18T14:23:43.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf38996700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:23:43.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf38996700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 0x7fbf34096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf38996700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf340646a0 con 0x7fbf3413fba0 2024-09-18T14:23:43.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 0x7fbf34096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient: get_auth_request con 0x7fbf3413fba0 auth_method 0 2024-09-18T14:23:43.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 10 monclient(hunting): handle_auth_done global_id 4837 payload 824 2024-09-18T14:23:43.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.149+0000 7fbf3259c700 1 -- 172.21.15.104:0/144279843 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbf34118900 con 0x7fbf3413fba0 2024-09-18T14:23:43.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf3259c700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 msgr2=0x7fbf34096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:43.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf3259c700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 0x7fbf34096a60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbf24019000 tx=0x7fbf2401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004bb0 con 0x7fbf3413fba0 2024-09-18T14:23:43.156 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 10 monclient: dump: 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.157 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.158 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.158 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fbf2401d430 con 0x7fbf3413fba0 2024-09-18T14:23:43.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:23:43.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:43.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 msgr2=0x7fbf34096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 0x7fbf34096a60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbf24019000 tx=0x7fbf2401a000 comp rx=0 tx=0).stop 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 msgr2=0x7fbf34096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf2a7fc700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf3413fba0 0x7fbf34096a60 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbf24019000 tx=0x7fbf2401a000 comp rx=0 tx=0).stop 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 20, 2024-09-18T14:23:43.159 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 2, 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:43.160 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:43.161 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:23:43.162 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:43.163 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.164 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:43.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:23:43.167 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:43.170 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 6 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:43.171 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:43.172 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:43.172 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:43.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:43.172 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:23:43.172 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 >> v2:172.21.15.174:6812/17603 conn(0x7fbf14035a80 msgr2=0x7fbf14037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 --2- 172.21.15.104:0/144279843 >> v2:172.21.15.174:6812/17603 conn(0x7fbf14035a80 0x7fbf14037f30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbf341386a0 tx=0x7fbf1c007ea0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 10 monclient: shutdown 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf341351b0 msgr2=0x7fbf3413f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 --2- 172.21.15.104:0/144279843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf341351b0 0x7fbf3413f120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0093f0 tx=0x7fbf2c016000 comp rx=0 tx=0).stop 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf32d9d700 1 -- 172.21.15.104:0/144279843 reap_dead start 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf32d9d700 1 -- 172.21.15.104:0/144279843 reap_dead start 2024-09-18T14:23:43.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 shutdown_connections 2024-09-18T14:23:43.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 >> 172.21.15.104:0/144279843 conn(0x7fbf34068f90 msgr2=0x7fbf3409b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:43.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 shutdown_connections 2024-09-18T14:23:43.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.153+0000 7fbf38996700 1 -- 172.21.15.104:0/144279843 wait complete. 2024-09-18T14:23:43.175 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:23:43.175 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T14:23:43.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.281+0000 7f8406692700 10 monclient: get_monmap_and_config 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.281+0000 7f8406692700 10 monclient: build_initial_monmap 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.281+0000 7f8406692700 10 monclient: monmap: 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:43.286731+0000 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:43.286731+0000 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:43.288 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 1 Processor -- start 2024-09-18T14:23:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 1 -- start start 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient: init 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8400131d00 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f84001320d0 0x7f840009ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient(hunting): picked mon.noname-a con 0x7f84001320d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8400136ef0 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8400099f10 con 0x7f8400136ef0 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840009b530 con 0x7f8400131d00 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f840009b810 con 0x7f84001320d0 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8406692700 10 monclient(hunting): authenticate will time out at 944.276925s 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f84001320d0 0x7f840009ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f84001320d0 0x7f840009ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53518/0 (socket says 172.21.15.104:53518) 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 -- 172.21.15.104:0/1348145531 learned_addr learned my addr 172.21.15.104:0/1348145531 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): get_auth_request con 0x7f84001320d0 auth_method 0 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient(hunting): handle_auth_done global_id 14136 payload 274 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 -- 172.21.15.104:0/1348145531 >> v1:172.21.15.174:6789/0 conn(0x7f8400131d00 legacy=0x7f840009e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 -- 172.21.15.104:0/1348145531 >> v1:172.21.15.104:6789/0 conn(0x7f8400136ef0 legacy=0x7f84001392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient: _finish_hunting 0 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 monclient: found mon.noname-a 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 -- 172.21.15.104:0/1348145531 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840009f480 con 0x7f84001320d0 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 -- 172.21.15.104:0/1348145531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84001320d0 msgr2=0x7f840009ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f8404e8f700 1 --2- 172.21.15.104:0/1348145531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84001320d0 0x7f840009ef40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f83f00099b0 tx=0x7f83f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595936427a90e49 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.285+0000 7f83f7fff700 1 -- 172.21.15.104:0/1348145531 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f000bb50 con 0x7f84001320d0 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: dump: 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.298 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: _finish_auth 0 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:43.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 5 monclient: authenticate success, global_id 14136 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 1 -- 172.21.15.104:0/1348145531 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f000bcb0 con 0x7f84001320d0 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 4 set_mon_vals no callback set 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 1 -- 172.21.15.104:0/1348145531 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0011670 con 0x7f84001320d0 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f83f7fff700 10 monclient: dump: 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.300 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 10 monclient: shutdown 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 -- 172.21.15.104:0/1348145531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84001320d0 msgr2=0x7f840009ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 --2- 172.21.15.104:0/1348145531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84001320d0 0x7f840009ef40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f83f00099b0 tx=0x7f83f0009250 comp rx=0 tx=0).stop 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 -- 172.21.15.104:0/1348145531 shutdown_connections 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 --2- 172.21.15.104:0/1348145531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84001320d0 0x7f840009ef40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 -- 172.21.15.104:0/1348145531 >> 172.21.15.104:0/1348145531 conn(0x7f84000a5330 msgr2=0x7f84000a5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 -- 172.21.15.104:0/1348145531 shutdown_connections 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 -- 172.21.15.104:0/1348145531 wait complete. 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 10 monclient: build_initial_monmap 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 10 monclient: monmap: 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:43.302 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 Processor -- start 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 1 -- start start 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.289+0000 7f8406692700 10 monclient: init 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:23:43.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 0x7f8400134990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): picked mon.noname-a con 0x7f8400131d00 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f84001314a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): picked mon.noname-c con 0x7f84001320d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 0x7f8400132ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): picked mon.noname-b con 0x7f8400136ef0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84001319e0 con 0x7f8400131d00 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84001346c0 con 0x7f8400136ef0 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840012d260 con 0x7f84001320d0 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8406692700 10 monclient(hunting): authenticate will time out at 944.284686s 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 0x7f8400134990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f84001314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 0x7f8400134990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53524/0 (socket says 172.21.15.104:53524) 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f84001314a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59350/0 (socket says 172.21.15.104:59350) 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 -- 172.21.15.104:0/3829859324 learned_addr learned my addr 172.21.15.104:0/3829859324 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): get_auth_request con 0x7f8400131d00 auth_method 0 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.305 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 10 monclient(hunting): get_auth_request con 0x7f84001320d0 auth_method 0 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.306 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 0x7f8400132ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 10 monclient(hunting): get_auth_request con 0x7f8400136ef0 auth_method 0 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405e91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient(hunting): handle_auth_done global_id 14139 payload 824 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 msgr2=0x7f84001314a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f84001314a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 msgr2=0x7f8400132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 0x7f8400132ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient: _finish_hunting 0 2024-09-18T14:23:43.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 monclient: found mon.noname-a 2024-09-18T14:23:43.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:43.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8405690700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f00094f0 con 0x7f8400131d00 2024-09-18T14:23:43.308 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.293+0000 7f8404e8f700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f84001314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:23:43.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 msgr2=0x7f8400134990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:43.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 0x7f8400134990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00c600 tx=0x7f83fc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc0117a0 con 0x7f8400131d00 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: dump: 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.314 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: _finish_auth 0 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 5 monclient: authenticate success, global_id 14139 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83fc011900 con 0x7f8400131d00 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 10 monclient: _renew_subs 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8400135ff0 con 0x7f8400131d00 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:43.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc011cc0 con 0x7f8400131d00 2024-09-18T14:23:43.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:43.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f6ffd700 4 set_mon_vals no callback set 2024-09-18T14:23:43.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: dump: 2024-09-18T14:23:43.316 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.316 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.316 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 10 monclient: _renew_subs 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8400130900 con 0x7f8400131d00 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 msgr2=0x7f8400134990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T14:23:43.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 msgr2=0x7f8400134990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 0x7f8400134990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00c600 tx=0x7f83fc007f60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 0x7f8400134990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00c600 tx=0x7f83fc007f60 comp rx=0 tx=0).stop 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: ms_handle_reset current mon [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 msgr2=0x7f8400134990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8400131d00 0x7f8400134990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f83e4005830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient(hunting): picked mon.c con 0x7f84001320d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 0x7f83e40081a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient(hunting): picked mon.b con 0x7f8400136ef0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83e4008b20 0x7f83e400afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient(hunting): picked mon.a con 0x7f83e4008b20 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e400ba50 con 0x7f83e4008b20 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e400bb90 con 0x7f8400136ef0 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e400bcd0 con 0x7f84001320d0 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f83f57fa700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8406692700 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f83e4005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 10 monclient(hunting): get_auth_request con 0x7f84001320d0 auth_method 0 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83e4008b20 0x7f83e400afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): get_auth_request con 0x7f83e4008b20 auth_method 0 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:43.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 0x7f83e40081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 10 monclient(hunting): get_auth_request con 0x7f8400136ef0 auth_method 0 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8404e8f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient(hunting): handle_auth_done global_id 14139 payload 293 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 msgr2=0x7f83e4005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f83e4005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 msgr2=0x7f83e40081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 0x7f83e40081a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient: _finish_hunting 0 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 monclient: found mon.a 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f83e400bfc0 con 0x7f83e4008b20 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8400136330 con 0x7f83e4008b20 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f800f240 con 0x7f83e4008b20 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 10 monclient: _finish_auth 0 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405e91700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.297+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f83e4005830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f8405e91700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83e4008b20 msgr2=0x7f83e400afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f8405e91700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83e4008b20 0x7f83e400afa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f83f800f430 tx=0x7f83f8002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8020070 con 0x7f83e4008b20 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 10 monclient: dump: 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f8006db0 con 0x7f83e4008b20 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f77fe700 4 set_mon_vals no callback set 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f83f800fbb0 con 0x7f83e4008b20 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 1 --2- 172.21.15.104:0/3829859324 >> v2:172.21.15.174:6812/17603 conn(0x7f8400131d00 0x7f83e40403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 4 ==== osd_map(123..123 src has 1..123) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f83f8046640 con 0x7f83e4008b20 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> v2:172.21.15.174:6812/17603 conn(0x7f8400131d00 0x7f83e40403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.301+0000 7f8405690700 10 monclient: get_auth_request con 0x7f8400131d00 auth_method 0 2024-09-18T14:23:43.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.305+0000 7f8405690700 1 --2- 172.21.15.104:0/3829859324 >> v2:172.21.15.174:6812/17603 conn(0x7f8400131d00 0x7f83e40403e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007fa0 tx=0x7f83fc017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:43.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.309+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f83f801b5c0 con 0x7f83e4008b20 2024-09-18T14:23:43.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.309+0000 7f83f57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:43.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.309+0000 7f83f57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:43.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.317+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f83f8012b40 con 0x7f83e4008b20 2024-09-18T14:23:43.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.317+0000 7f83f57fa700 10 monclient: handle_mon_command_ack 1 not found 2024-09-18T14:23:43.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.545+0000 7f8406692700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T14:23:43.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.545+0000 7f8406692700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T14:23:43.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.545+0000 7f8406692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:43.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.545+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8400136330 con 0x7f83e4008b20 2024-09-18T14:23:43.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f83f57fa700 1 -- 172.21.15.104:0/3829859324 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f8400136330 con 0x7f83e4008b20 2024-09-18T14:23:43.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f83f57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T14:23:43.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f83f57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:43.553 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.553 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=8,logm=70} crc {auth=1234867316,config=3140514009,health=3977944326,kv=4078166765,logm=1030226798}) 2024-09-18T14:23:43.553 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=8,logm=70} crc {auth=1234867316,config=3140514009,health=3977944326,kv=4078166765,logm=4050414128}) 2024-09-18T14:23:43.553 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.553 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=8,logm=70} crc {auth=1234867316,config=3140514009,health=3977944326,kv=4078166765,logm=1030226798}) 2024-09-18T14:23:43.553 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=12,kv=8,logm=70} crc {auth=1234867316,config=3140514009,health=3977944326,kv=4078166765,logm=3778204808}) 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 >> v2:172.21.15.174:6812/17603 conn(0x7f8400131d00 msgr2=0x7f83e40403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 --2- 172.21.15.104:0/3829859324 >> v2:172.21.15.174:6812/17603 conn(0x7f8400131d00 0x7f83e40403e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007fa0 tx=0x7f83fc017000 comp rx=0 tx=0).stop 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 10 monclient: shutdown 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83e4008b20 msgr2=0x7f83e400afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83e4008b20 0x7f83e400afa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f83f800f430 tx=0x7f83f8002620 comp rx=0 tx=0).stop 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 shutdown_connections 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 --2- 172.21.15.104:0/3829859324 >> v2:172.21.15.174:6812/17603 conn(0x7f8400131d00 0x7f83e40403e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83e4008b20 0x7f83e400afa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8400136ef0 0x7f83e40081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 --2- 172.21.15.104:0/3829859324 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f84001320d0 0x7f83e4005830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:43.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 >> 172.21.15.104:0/3829859324 conn(0x7f84000a5330 msgr2=0x7f840012bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:43.556 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 shutdown_connections 2024-09-18T14:23:43.556 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439875058}) 2024-09-18T14:23:43.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:43.549+0000 7f8406692700 1 -- 172.21.15.104:0/3829859324 wait complete. 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2126697528}) 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439875058}) 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.549+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403971610}) 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.553+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.553+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427869292}) 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.553+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=417997735}) 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.553+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.553+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427869292}) 2024-09-18T14:23:43.558 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.553+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3471986961}) 2024-09-18T14:23:43.560 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.560 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120411893}) 2024-09-18T14:23:43.560 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515665069}) 2024-09-18T14:23:43.560 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.560 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120411893}) 2024-09-18T14:23:43.560 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=287408395}) 2024-09-18T14:23:43.562 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.562 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958998579}) 2024-09-18T14:23:43.562 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489922317}) 2024-09-18T14:23:43.562 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.562 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958998579}) 2024-09-18T14:23:43.562 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.557+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457577845}) 2024-09-18T14:23:43.563 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.564 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=4163948119,mdsmap=1531151339,mgr=3617533101,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=1715951778}) 2024-09-18T14:23:43.564 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_metadata=1} crc {logm=233368743,mdsmap=1531151339,mgr=3617533101,mgr_metadata=2046822578}) 2024-09-18T14:23:43.564 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.564 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=4163948119,mdsmap=1531151339,mgr=3617533101,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=1715951778}) 2024-09-18T14:23:43.564 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=85,mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3568874583,mdsmap=1228700967,mgr=1292546760,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=73008305}) 2024-09-18T14:23:43.565 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.565 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=4104878239,mon_config_key=1784637070,monmap=530371891,osd_metadata=2265752427,osd_pg_creating=139760628,osd_snap=3178633393,osdmap=725122602}) 2024-09-18T14:23:43.565 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mgrstat=3021651602,mon_config_key=1784637070,monmap=530371891,osd_metadata=2265752427,osd_pg_creating=139760628,osd_snap=3178633393,osdmap=346586387}) 2024-09-18T14:23:43.565 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.565 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=4104878239,mon_config_key=1784637070,monmap=530371891,osd_metadata=2265752427,osd_pg_creating=139760628,osd_snap=3178633393,osdmap=725122602}) 2024-09-18T14:23:43.565 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mgrstat=4104878239,mon_config_key=1401738264,monmap=530371891,osd_metadata=1315867136,osd_pg_creating=139760628,osd_snap=3178633393,osdmap=2195738989}) 2024-09-18T14:23:43.566 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.566 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3510560132}) 2024-09-18T14:23:43.566 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2405725331}) 2024-09-18T14:23:43.566 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.567 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3510560132}) 2024-09-18T14:23:43.567 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3742441995}) 2024-09-18T14:23:43.567 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.567 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=1690966470}) 2024-09-18T14:23:43.567 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=4016763256}) 2024-09-18T14:23:43.567 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:23:43.567 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=1690966470}) 2024-09-18T14:23:43.568 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:23:43.561+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2040435839}) 2024-09-18T14:23:43.568 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T14:23:43.631 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19732]: starting. 2024-09-18T14:23:43.631 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19732]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19731 2024-09-18T14:23:43.631 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19732]: created object 0... 2024-09-18T14:23:43.631 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19732]: created object 25... 2024-09-18T14:23:43.631 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19732]: created object 49... 2024-09-18T14:23:43.631 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19732]: finishing. 2024-09-18T14:23:43.631 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19732]: shutting down. 2024-09-18T14:23:44.488 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19733]: starting. 2024-09-18T14:23:44.489 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19733]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19731 2024-09-18T14:23:44.489 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19733]: shutting down. 2024-09-18T14:23:44.569 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T14:23:44.570 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:44.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient: get_monmap_and_config 2024-09-18T14:23:44.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient: build_initial_monmap 2024-09-18T14:23:44.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient: monmap: 2024-09-18T14:23:44.672 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:44.672 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:44.672220+0000 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:44.672220+0000 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:44.673 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:44.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 1 Processor -- start 2024-09-18T14:23:44.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 1 -- start start 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient: init 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f207c136200 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f207c1365d0 0x7f207c09ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient(hunting): picked mon.noname-a con 0x7f207c1365d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f207c13b390 0x7f207c13d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient(hunting): picked mon.noname-c con 0x7f207c13b390 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c095cf0 con 0x7f207c136200 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207c095b20 con 0x7f207c1365d0 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:44.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f207c0970b0 con 0x7f207c13b390 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f2080d29700 10 monclient(hunting): authenticate will time out at 945.661805s 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f207c1365d0 0x7f207c09ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f207c1365d0 0x7f207c09ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53532/0 (socket says 172.21.15.104:53532) 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 -- 172.21.15.104:0/1156540758 learned_addr learned my addr 172.21.15.104:0/1156540758 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): get_auth_request con 0x7f207c1365d0 auth_method 0 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 1 --2- 172.21.15.104:0/1156540758 >> v2:172.21.15.174:3300/0 conn(0x7f207c13b390 0x7f207c13d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 10 monclient(hunting): get_auth_request con 0x7f207c13b390 auth_method 0 2024-09-18T14:23:44.677 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient(hunting): handle_auth_done global_id 14157 payload 274 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 -- 172.21.15.104:0/1156540758 >> v2:172.21.15.174:3300/0 conn(0x7f207c13b390 msgr2=0x7f207c13d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 --2- 172.21.15.104:0/1156540758 >> v2:172.21.15.174:3300/0 conn(0x7f207c13b390 0x7f207c13d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 -- 172.21.15.104:0/1156540758 >> v1:172.21.15.174:6789/0 conn(0x7f207c136200 legacy=0x7f207c09a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient: _finish_hunting 0 2024-09-18T14:23:44.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 monclient: found mon.noname-a 2024-09-18T14:23:44.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:44.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:44.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.669+0000 7f207affd700 1 -- 172.21.15.104:0/1156540758 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c097390 con 0x7f207c1365d0 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207affd700 1 -- 172.21.15.104:0/1156540758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 msgr2=0x7f207c09ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207affd700 1 --2- 172.21.15.104:0/1156540758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c09ab30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f207c096fe0 tx=0x7f206c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3d996f5955cbbf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 1 -- 172.21.15.104:0/1156540758 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c00e070 con 0x7f207c1365d0 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: dump: 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:44.682 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 1 -- 172.21.15.104:0/1156540758 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206c00bc30 con 0x7f207c1365d0 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 5 monclient: authenticate success, global_id 14157 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 4 set_mon_vals no callback set 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 1 -- 172.21.15.104:0/1156540758 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c0092e0 con 0x7f207c1365d0 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f207a7fc700 10 monclient: dump: 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 10 monclient: shutdown 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 -- 172.21.15.104:0/1156540758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 msgr2=0x7f207c09ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 --2- 172.21.15.104:0/1156540758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c09ab30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f207c096fe0 tx=0x7f206c009840 comp rx=0 tx=0).stop 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 -- 172.21.15.104:0/1156540758 shutdown_connections 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 --2- 172.21.15.104:0/1156540758 >> v2:172.21.15.174:3300/0 conn(0x7f207c13b390 0x7f207c13d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 --2- 172.21.15.104:0/1156540758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c09ab30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 -- 172.21.15.104:0/1156540758 >> 172.21.15.104:0/1156540758 conn(0x7f207c0a12c0 msgr2=0x7f207c0a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 -- 172.21.15.104:0/1156540758 shutdown_connections 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 1 -- 172.21.15.104:0/1156540758 wait complete. 2024-09-18T14:23:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 10 monclient: build_initial_monmap 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.673+0000 7f2080d29700 10 monclient: monmap: 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 Processor -- start 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 -- start start 2024-09-18T14:23:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient: init 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 0x7f207c134ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): picked mon.noname-c con 0x7f207c136200 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): picked mon.noname-a con 0x7f207c1365d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c13b390 0x7f207c06df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): picked mon.noname-b con 0x7f207c13b390 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c09fde0 con 0x7f207c1365d0 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207c121630 con 0x7f207c13b390 2024-09-18T14:23:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207c13bc70 con 0x7f207c136200 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f2080d29700 10 monclient(hunting): authenticate will time out at 945.668951s 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 0x7f207c134ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c06da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53538/0 (socket says 172.21.15.104:53538) 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 0x7f207c134ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59374/0 (socket says 172.21.15.104:59374) 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 1 -- 172.21.15.104:0/1435081858 learned_addr learned my addr 172.21.15.104:0/1435081858 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 10 monclient(hunting): get_auth_request con 0x7f207c1365d0 auth_method 0 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): get_auth_request con 0x7f207c136200 auth_method 0 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c13b390 0x7f207c06df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 10 monclient(hunting): get_auth_request con 0x7f207c13b390 auth_method 0 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:44.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient(hunting): handle_auth_done global_id 4829 payload 824 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 -- 172.21.15.104:0/1435081858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c13b390 msgr2=0x7f207c06df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c13b390 0x7f207c06df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 -- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 msgr2=0x7f207c06da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c06da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 monclient: found mon.noname-c 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:23:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207b7fe700 1 -- 172.21.15.104:0/1435081858 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c009ae0 con 0x7f207c136200 2024-09-18T14:23:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.677+0000 7f207affd700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c06da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f207b7fe700 1 -- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 msgr2=0x7f207c134ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f207b7fe700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 0x7f207c134ed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f207000c4e0 tx=0x7f2070007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 1 -- 172.21.15.104:0/1435081858 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207000ce90 con 0x7f207c136200 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: dump: 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: _finish_auth 0 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 5 monclient: authenticate success, global_id 4829 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 10 monclient: _renew_subs 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 1 -- 172.21.15.104:0/1435081858 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20700117a0 con 0x7f207c136200 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207c06f6c0 con 0x7f207c136200 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:44.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 1 -- 172.21.15.104:0/1435081858 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070011b40 con 0x7f207c136200 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 10 monclient: dump: 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:44.696 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 10 monclient: _renew_subs 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207c1212c0 con 0x7f207c136200 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2079ffb700 4 set_mon_vals no callback set 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 1 -- 172.21.15.104:0/1435081858 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f207001c460 con 0x7f207c136200 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207c0633b0 con 0x7f207c136200 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 1 --2- 172.21.15.104:0/1435081858 >> v2:172.21.15.174:6812/17603 conn(0x7f2068035a80 0x7f2068037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:44.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f20637fe700 1 -- 172.21.15.104:0/1435081858 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2070047930 con 0x7f207c136200 2024-09-18T14:23:44.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f207affd700 1 --2- 172.21.15.104:0/1435081858 >> v2:172.21.15.174:6812/17603 conn(0x7f2068035a80 0x7f2068037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:44.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.681+0000 7f207affd700 10 monclient: get_auth_request con 0x7f2068035a80 auth_method 0 2024-09-18T14:23:44.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.685+0000 7f207affd700 1 --2- 172.21.15.104:0/1435081858 >> v2:172.21.15.174:6812/17603 conn(0x7f2068035a80 0x7f2068037f30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f206c009930 tx=0x7f206c017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:44.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.689+0000 7f20637fe700 1 -- 172.21.15.104:0/1435081858 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2070012420 con 0x7f207c136200 2024-09-18T14:23:44.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.689+0000 7f20637fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:44.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.689+0000 7f20637fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:44.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.909+0000 7f2080d29700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:44.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.909+0000 7f2080d29700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:44.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.909+0000 7f2080d29700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:44.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.909+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f207c06f660 con 0x7f207c136200 2024-09-18T14:23:44.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f20637fe700 1 -- 172.21.15.104:0/1435081858 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2070019310 con 0x7f207c136200 2024-09-18T14:23:44.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f20637fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:44.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f20637fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:44.917 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:44.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 >> v2:172.21.15.174:6812/17603 conn(0x7f2068035a80 msgr2=0x7f2068037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:44.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 --2- 172.21.15.104:0/1435081858 >> v2:172.21.15.174:6812/17603 conn(0x7f2068035a80 0x7f2068037f30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f206c009930 tx=0x7f206c017000 comp rx=0 tx=0).stop 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 10 monclient: shutdown 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 msgr2=0x7f207c134ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 0x7f207c134ed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f207000c4e0 tx=0x7f2070007ea0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 shutdown_connections 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 --2- 172.21.15.104:0/1435081858 >> v2:172.21.15.174:6812/17603 conn(0x7f2068035a80 0x7f2068037f30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c13b390 0x7f207c06df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f207c1365d0 0x7f207c06da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 --2- 172.21.15.104:0/1435081858 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f207c136200 0x7f207c134ed0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:44.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 >> 172.21.15.104:0/1435081858 conn(0x7f207c0a12c0 msgr2=0x7f207c098d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:44.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 shutdown_connections 2024-09-18T14:23:44.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:44.913+0000 7f2080d29700 1 -- 172.21.15.104:0/1435081858 wait complete. 2024-09-18T14:23:44.930 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T14:23:44.930 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T14:23:44.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T14:23:44.930 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:23:45.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.029+0000 7f8ec4445700 10 monclient: get_monmap_and_config 2024-09-18T14:23:45.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.029+0000 7f8ec4445700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.029+0000 7f8ec4445700 10 monclient: monmap: 2024-09-18T14:23:45.035 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.035 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.035 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:45.034978+0000 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:45.034978+0000 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.029+0000 7f8ec4445700 1 Processor -- start 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.029+0000 7f8ec4445700 1 -- start start 2024-09-18T14:23:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.029+0000 7f8ec4445700 10 monclient: init 2024-09-18T14:23:45.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T14:23:45.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ebc097c60 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:45.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ebc09adf0 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:45.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09b1c0 0x7f8ebc1385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient(hunting): picked mon.noname-b con 0x7f8ebc09b1c0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc09db70 con 0x7f8ebc097c60 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc099300 con 0x7f8ebc09adf0 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc130260 con 0x7f8ebc09b1c0 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec4445700 10 monclient(hunting): authenticate will time out at 946.024819s 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09b1c0 0x7f8ebc1385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec21e1700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8ebc097c60 0x7f8ebc12fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59810/0 (socket says 172.21.15.104:59810) 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): get_auth_request con 0x7f8ebc09b1c0 auth_method 0 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec21e1700 1 -- 172.21.15.104:0/4180717384 learned_addr learned my addr 172.21.15.104:0/4180717384 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient(hunting): handle_auth_done global_id 4832 payload 274 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 1 -- 172.21.15.104:0/4180717384 >> v1:172.21.15.174:6789/0 conn(0x7f8ebc09adf0 legacy=0x7f8ebc134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 1 -- 172.21.15.104:0/4180717384 >> v1:172.21.15.104:6789/0 conn(0x7f8ebc097c60 legacy=0x7f8ebc12fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 1 monclient: found mon.noname-b 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 1 -- 172.21.15.104:0/4180717384 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc098030 con 0x7f8ebc09b1c0 2024-09-18T14:23:45.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 1 -- 172.21.15.104:0/4180717384 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09b1c0 msgr2=0x7f8ebc1385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.033+0000 7f8ec29e2700 1 --2- 172.21.15.104:0/4180717384 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09b1c0 0x7f8ebc1385b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009e90 tx=0x7f8eb4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b625f0c1912d6476 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 1 -- 172.21.15.104:0/4180717384 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400e070 con 0x7f8ebc09b1c0 2024-09-18T14:23:45.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:23:45.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: dump: 2024-09-18T14:23:45.044 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 1 -- 172.21.15.104:0/4180717384 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb4012430 con 0x7f8ebc09b1c0 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 5 monclient: authenticate success, global_id 4832 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 4 set_mon_vals no callback set 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 1 -- 172.21.15.104:0/4180717384 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40127f0 con 0x7f8ebc09b1c0 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:45.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec11df700 10 monclient: dump: 2024-09-18T14:23:45.047 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.047 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.048 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.048 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.048 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.048 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.048 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.048 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient: shutdown 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- 172.21.15.104:0/4180717384 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09b1c0 msgr2=0x7f8ebc1385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 --2- 172.21.15.104:0/4180717384 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09b1c0 0x7f8ebc1385b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009e90 tx=0x7f8eb4009780 comp rx=0 tx=0).stop 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- 172.21.15.104:0/4180717384 shutdown_connections 2024-09-18T14:23:45.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 --2- 172.21.15.104:0/4180717384 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09b1c0 0x7f8ebc1385b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- 172.21.15.104:0/4180717384 >> 172.21.15.104:0/4180717384 conn(0x7f8ebc0a1290 msgr2=0x7f8ebc0a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- 172.21.15.104:0/4180717384 shutdown_connections 2024-09-18T14:23:45.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- 172.21.15.104:0/4180717384 wait complete. 2024-09-18T14:23:45.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient: monmap: 2024-09-18T14:23:45.050 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.051 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:45.051 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:45.051 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:45.051 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.051 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.051 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:45.051 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 Processor -- start 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- start start 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient: init 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc097c60 0x7f8ebc12f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): picked mon.noname-a con 0x7f8ebc097c60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09adf0 0x7f8ebc12b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): picked mon.noname-c con 0x7f8ebc09adf0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ebc09b1c0 0x7f8ebc12bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): picked mon.noname-b con 0x7f8ebc09b1c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc12b340 con 0x7f8ebc097c60 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc098770 con 0x7f8ebc09b1c0 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc12e6d0 con 0x7f8ebc09adf0 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec4445700 10 monclient(hunting): authenticate will time out at 946.030607s 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09adf0 0x7f8ebc12b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09adf0 0x7f8ebc12b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59400/0 (socket says 172.21.15.104:59400) 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 -- 172.21.15.104:0/1882124650 learned_addr learned my addr 172.21.15.104:0/1882124650 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): get_auth_request con 0x7f8ebc09adf0 auth_method 0 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec21e1700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc097c60 0x7f8ebc12f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec29e2700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ebc09b1c0 0x7f8ebc12bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec21e1700 10 monclient(hunting): get_auth_request con 0x7f8ebc097c60 auth_method 0 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec21e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec21e1700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec21e1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec29e2700 10 monclient(hunting): get_auth_request con 0x7f8ebc09b1c0 auth_method 0 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec29e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec29e2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec29e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient(hunting): handle_auth_done global_id 4835 payload 824 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ebc09b1c0 msgr2=0x7f8ebc12bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ebc09b1c0 0x7f8ebc12bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc097c60 msgr2=0x7f8ebc12f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc097c60 0x7f8ebc12f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 monclient: found mon.noname-c 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec19e0700 1 -- 172.21.15.104:0/1882124650 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4009a20 con 0x7f8ebc09adf0 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec29e2700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ebc09b1c0 0x7f8ebc12bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:23:45.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.037+0000 7f8ec21e1700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc097c60 0x7f8ebc12f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:23:45.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec19e0700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09adf0 msgr2=0x7f8ebc12b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec19e0700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09adf0 0x7f8ebc12b6f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800c600 tx=0x7f8eb8007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb80117a0 con 0x7f8ebc09adf0 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: dump: 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.060 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 5 monclient: authenticate success, global_id 4835 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb8011900 con 0x7f8ebc09adf0 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 10 monclient: _renew_subs 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc12e450 con 0x7f8ebc09adf0 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8011cc0 con 0x7f8ebc09adf0 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb3fff700 4 set_mon_vals no callback set 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 10 monclient: dump: 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.062 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 10 monclient: _renew_subs 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc12c170 con 0x7f8ebc09adf0 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f8eb8012630 con 0x7f8ebc09adf0 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 0x7f8ebc13f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc068690 con 0x7f8ebc12f5f0 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec21e1700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 0x7f8ebc13f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8eb27fc700 1 --2- 172.21.15.104:0/1882124650 >> v2:172.21.15.174:6812/17603 conn(0x7f8ea0035a80 0x7f8ea0037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.041+0000 7f8ec21e1700 10 monclient: get_auth_request con 0x7f8ebc12f5f0 auth_method 0 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8eb8047cc0 con 0x7f8ebc09adf0 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_done global_id 14166 payload 824 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec29e2700 1 --2- 172.21.15.104:0/1882124650 >> v2:172.21.15.174:6812/17603 conn(0x7f8ea0035a80 0x7f8ea0037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 1 -- 172.21.15.104:0/1882124650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ebc12c930 con 0x7f8ebc12f5f0 2024-09-18T14:23:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec29e2700 10 monclient: get_auth_request con 0x7f8ea0035a80 auth_method 0 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 msgr2=0x7f8ebc13f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec21e1700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 0x7f8ebc13f1d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00d740 tx=0x7f8eac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac0129c0 con 0x7f8ebc12f5f0 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 10 monclient: dump: 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8ec29e2700 1 --2- 172.21.15.104:0/1882124650 >> v2:172.21.15.174:6812/17603 conn(0x7f8ea0035a80 0x7f8ea0037f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40099f0 tx=0x7f8eb4018000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8eac012b20 con 0x7f8ebc12f5f0 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 msgr2=0x7f8ebc13f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 0x7f8ebc13f1d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00d740 tx=0x7f8eac008040 comp rx=0 tx=0).stop 2024-09-18T14:23:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 msgr2=0x7f8ebc13f1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:45.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.045+0000 7f8eb27fc700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc12f5f0 0x7f8ebc13f1d0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00d740 tx=0x7f8eac008040 comp rx=0 tx=0).stop 2024-09-18T14:23:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec4445700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:23:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec4445700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 0x7f8ebc13fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec4445700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc140050 con 0x7f8ebc13f710 2024-09-18T14:23:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 0x7f8ebc13fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient: get_auth_request con 0x7f8ebc13f710 auth_method 0 2024-09-18T14:23:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 10 monclient(hunting): handle_auth_done global_id 14169 payload 824 2024-09-18T14:23:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.057+0000 7f8ec21e1700 1 -- 172.21.15.104:0/1882124650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8ebc064b30 con 0x7f8ebc13f710 2024-09-18T14:23:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec21e1700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 msgr2=0x7f8ebc13fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec21e1700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 0x7f8ebc13fb10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8eac018000 tx=0x7f8eac01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac0209e0 con 0x7f8ebc13f710 2024-09-18T14:23:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 10 monclient: dump: 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.074 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f8eac020b40 con 0x7f8ebc13f710 2024-09-18T14:23:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:23:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 msgr2=0x7f8ebc13fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 0x7f8ebc13fb10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8eac018000 tx=0x7f8eac01b000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 msgr2=0x7f8ebc13fb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8eb27fc700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ebc13f710 0x7f8ebc13fb10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.076 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:23:45.076 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:45.076 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:45.076 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:23:45.076 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 20, 2024-09-18T14:23:45.077 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:23:45.077 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:23:45.077 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:23:45.077 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:23:45.077 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.078 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 4, 2024-09-18T14:23:45.078 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:45.078 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:23:45.078 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:23:45.078 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.078 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.078 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.080 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:23:45.080 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:23:45.080 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.080 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.080 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.080 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.080 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.081 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.081 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.081 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.081 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.081 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.081 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:23:45.082 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:23:45.082 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:23:45.082 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:23:45.082 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:23:45.082 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:23:45.082 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:45.083 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:45.083 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:23:45.083 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:23:45.083 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:23:45.083 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:23:45.083 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.084 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:45.085 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.087 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.088 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.089 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.090 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:23:45.092 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:23:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 >> v2:172.21.15.174:6812/17603 conn(0x7f8ea0035a80 msgr2=0x7f8ea0037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 --2- 172.21.15.104:0/1882124650 >> v2:172.21.15.174:6812/17603 conn(0x7f8ea0035a80 0x7f8ea0037f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40099f0 tx=0x7f8eb4018000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 10 monclient: shutdown 2024-09-18T14:23:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09adf0 msgr2=0x7f8ebc12b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 --2- 172.21.15.104:0/1882124650 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8ebc09adf0 0x7f8ebc12b6f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800c600 tx=0x7f8eb8007f60 comp rx=0 tx=0).stop 2024-09-18T14:23:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec29e2700 1 -- 172.21.15.104:0/1882124650 reap_dead start 2024-09-18T14:23:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec29e2700 1 -- 172.21.15.104:0/1882124650 reap_dead start 2024-09-18T14:23:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 shutdown_connections 2024-09-18T14:23:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 >> 172.21.15.104:0/1882124650 conn(0x7f8ebc0a1290 msgr2=0x7f8ebc138000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 shutdown_connections 2024-09-18T14:23:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.061+0000 7f8ec4445700 1 -- 172.21.15.104:0/1882124650 wait complete. 2024-09-18T14:23:45.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:23:45.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T14:23:45.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T14:23:45.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient: get_monmap_and_config 2024-09-18T14:23:45.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient: monmap: 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:45.205571+0000 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:45.205571+0000 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:45.206 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:45.207 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:45.207 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 1 Processor -- start 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 1 -- start start 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient: init 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb809e830 0x7fafb8131d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient(hunting): picked mon.noname-b con 0x7fafb809e830 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafb8132260 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fafb8136e60 0x7fafb8139240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient(hunting): picked mon.noname-a con 0x7fafb8136e60 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb809b7a0 con 0x7fafb8132260 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb8099f10 con 0x7fafb8136e60 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb809b4c0 con 0x7fafb809e830 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafbc83b700 10 monclient(hunting): authenticate will time out at 946.195496s 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fafb8136e60 0x7fafb8139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb77fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb809e830 0x7fafb8131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fafb8136e60 0x7fafb8139240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53560/0 (socket says 172.21.15.104:53560) 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 1 -- 172.21.15.104:0/3040831785 learned_addr learned my addr 172.21.15.104:0/3040831785 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 10 monclient(hunting): get_auth_request con 0x7fafb8136e60 auth_method 0 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb6ffd700 1 --1- 172.21.15.104:0/3040831785 >> v1:172.21.15.104:6789/0 conn(0x7fafb8132260 0x7fafb809ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59814/0 (socket says 172.21.15.104:59814) 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb77fe700 10 monclient(hunting): get_auth_request con 0x7fafb809e830 auth_method 0 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.201+0000 7fafb7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 10 monclient(hunting): handle_auth_done global_id 14172 payload 274 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 1 -- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb809e830 msgr2=0x7fafb8131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 1 --2- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb809e830 0x7fafb8131d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 1 -- 172.21.15.104:0/3040831785 >> v1:172.21.15.104:6789/0 conn(0x7fafb8132260 legacy=0x7fafb809ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:45.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 1 monclient: found mon.noname-a 2024-09-18T14:23:45.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 1 -- 172.21.15.104:0/3040831785 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb809ba80 con 0x7fafb8136e60 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 1 -- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 msgr2=0x7fafb8139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb7fff700 1 --2- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb8139240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fafb809be30 tx=0x7fafb0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb8105a5194d3e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 1 -- 172.21.15.104:0/3040831785 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb000e070 con 0x7fafb8136e60 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: dump: 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.218 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 5 monclient: authenticate success, global_id 14172 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 1 -- 172.21.15.104:0/3040831785 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb000bc30 con 0x7fafb8136e60 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:45.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 4 set_mon_vals no callback set 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 1 -- 172.21.15.104:0/3040831785 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00092e0 con 0x7fafb8136e60 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafb67fc700 10 monclient: dump: 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.220 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 10 monclient: shutdown 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 -- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 msgr2=0x7fafb8139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 --2- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb8139240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fafb809be30 tx=0x7fafb0009840 comp rx=0 tx=0).stop 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 -- 172.21.15.104:0/3040831785 shutdown_connections 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 --2- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb8139240 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 --2- 172.21.15.104:0/3040831785 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb809e830 0x7fafb8131d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 -- 172.21.15.104:0/3040831785 >> 172.21.15.104:0/3040831785 conn(0x7fafb80676a0 msgr2=0x7fafb8067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 -- 172.21.15.104:0/3040831785 shutdown_connections 2024-09-18T14:23:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.205+0000 7fafbc83b700 1 -- 172.21.15.104:0/3040831785 wait complete. 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient: monmap: 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 Processor -- start 2024-09-18T14:23:45.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 -- start start 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient: init 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb809e830 0x7fafb806da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): picked mon.noname-b con 0x7fafb809e830 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb8132260 0x7fafb806df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): picked mon.noname-c con 0x7fafb8132260 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb806e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): picked mon.noname-a con 0x7fafb8136e60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:45.223 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb809f5e0 con 0x7fafb8136e60 2024-09-18T14:23:45.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80a41d0 con 0x7fafb809e830 2024-09-18T14:23:45.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb811d2b0 con 0x7fafb8132260 2024-09-18T14:23:45.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafbc83b700 10 monclient(hunting): authenticate will time out at 946.202229s 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb8132260 0x7fafb806df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb806e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb8132260 0x7fafb806df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59412/0 (socket says 172.21.15.104:59412) 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb806e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53570/0 (socket says 172.21.15.104:53570) 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 1 -- 172.21.15.104:0/2816947017 learned_addr learned my addr 172.21.15.104:0/2816947017 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): get_auth_request con 0x7fafb8136e60 auth_method 0 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 10 monclient(hunting): get_auth_request con 0x7fafb8132260 auth_method 0 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb809e830 0x7fafb806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 10 monclient(hunting): get_auth_request con 0x7fafb809e830 auth_method 0 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient(hunting): handle_auth_done global_id 14175 payload 824 2024-09-18T14:23:45.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb8132260 msgr2=0x7fafb806df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb8132260 0x7fafb806df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb809e830 msgr2=0x7fafb806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafb809e830 0x7fafb806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 monclient: found mon.noname-a 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.209+0000 7fafb7fff700 1 -- 172.21.15.104:0/2816947017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0009ae0 con 0x7fafb8136e60 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafb7fff700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 msgr2=0x7fafb806e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafb7fff700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb806e4a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fafb8069820 tx=0x7fafb0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0007040 con 0x7fafb8136e60 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:45.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: dump: 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb0021070 con 0x7fafb8136e60 2024-09-18T14:23:45.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 5 monclient: authenticate success, global_id 14175 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 10 monclient: _renew_subs 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb806f3b0 con 0x7fafb8136e60 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 10 monclient: _renew_subs 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb811ba40 con 0x7fafb8136e60 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafb57fa700 4 set_mon_vals no callback set 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0007bf0 con 0x7fafb8136e60 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 10 monclient: dump: 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.231 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.232 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafb7fff700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb806e4a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fafb8069820 tx=0x7fafb0018000 comp rx=0 tx=0).queue_received will delay after 1.72667e+09 on 0x7fafb0049b50 osd_map(125..125 src has 1..125) v4 2024-09-18T14:23:45.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fafb0023030 con 0x7fafb8136e60 2024-09-18T14:23:45.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafaf7fe700 1 --2- 172.21.15.104:0/2816947017 >> v2:172.21.15.174:6812/17603 conn(0x7faf9c035a80 0x7faf9c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 0x7fafb802ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8064840 con 0x7fafb806e9e0 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.213+0000 7fafb6ffd700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 0x7fafb802ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient: get_auth_request con 0x7fafb806e9e0 auth_method 0 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb77fe700 1 --2- 172.21.15.104:0/2816947017 >> v2:172.21.15.174:6812/17603 conn(0x7faf9c035a80 0x7faf9c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb77fe700 10 monclient: get_auth_request con 0x7faf9c035a80 auth_method 0 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_done global_id 4844 payload 824 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 1 -- 172.21.15.104:0/2816947017 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafb811ccd0 con 0x7fafb806e9e0 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 msgr2=0x7fafb802ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb6ffd700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 0x7fafb802ae80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fafa00078b0 tx=0x7fafa0007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0005c80 con 0x7fafb806e9e0 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 10 monclient: dump: 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.237 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafb77fe700 1 --2- 172.21.15.104:0/2816947017 >> v2:172.21.15.174:6812/17603 conn(0x7faf9c035a80 0x7faf9c037f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009db0 tx=0x7fafa80091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fafa0005de0 con 0x7fafb806e9e0 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 msgr2=0x7fafb802ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 0x7fafb802ae80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fafa00078b0 tx=0x7fafa0007f90 comp rx=0 tx=0).stop 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 msgr2=0x7fafb802ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.217+0000 7fafaf7fe700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb806e9e0 0x7fafb802ae80 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fafa00078b0 tx=0x7fafa0007f90 comp rx=0 tx=0).stop 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafbc83b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafbc83b700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 0x7fafb805ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafbc83b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8136040 con 0x7fafb805e890 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 0x7fafb805ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 10 monclient: get_auth_request con 0x7fafb805e890 auth_method 0 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.229+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafb6ffd700 10 monclient(hunting): handle_auth_done global_id 4847 payload 824 2024-09-18T14:23:45.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafb6ffd700 1 -- 172.21.15.104:0/2816947017 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fafb8063450 con 0x7fafb805e890 2024-09-18T14:23:45.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafb6ffd700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 msgr2=0x7fafb805ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafb6ffd700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 0x7fafb805ec90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafa0017000 tx=0x7fafa0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0013cf0 con 0x7fafb805e890 2024-09-18T14:23:45.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 10 monclient: dump: 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fafa0013e50 con 0x7fafb805e890 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:23:45.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 msgr2=0x7fafb805ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 0x7fafb805ec90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafa0017000 tx=0x7fafa0018000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 msgr2=0x7fafb805ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.233+0000 7fafaf7fe700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafb805e890 0x7fafb805ec90 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafa0017000 tx=0x7fafa0018000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 20, 2024-09-18T14:23:45.242 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 4, 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.243 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.244 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:23:45.245 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.246 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.247 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.248 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.249 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:45.250 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.252 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.253 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:23:45.254 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:23:45.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 >> v2:172.21.15.174:6812/17603 conn(0x7faf9c035a80 msgr2=0x7faf9c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 --2- 172.21.15.104:0/2816947017 >> v2:172.21.15.174:6812/17603 conn(0x7faf9c035a80 0x7faf9c037f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009db0 tx=0x7fafa80091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 10 monclient: shutdown 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 msgr2=0x7fafb806e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 --2- 172.21.15.104:0/2816947017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb8136e60 0x7fafb806e4a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fafb8069820 tx=0x7fafb0018000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafb7fff700 1 -- 172.21.15.104:0/2816947017 reap_dead start 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafb7fff700 1 -- 172.21.15.104:0/2816947017 reap_dead start 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafaf7fe700 1 -- 172.21.15.104:0/2816947017 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (unknown 0 0 0) 0x7fafb0049b50 con 0x7fafb8136e60 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 shutdown_connections 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 >> 172.21.15.104:0/2816947017 conn(0x7fafb80676a0 msgr2=0x7fafb809d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 shutdown_connections 2024-09-18T14:23:45.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.237+0000 7fafbc83b700 1 -- 172.21.15.104:0/2816947017 wait complete. 2024-09-18T14:23:45.257 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:23:45.257 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T14:23:45.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:23:45.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient: get_monmap_and_config 2024-09-18T14:23:45.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient: monmap: 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:45.376623+0000 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:45.376623+0000 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:45.377 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 1 Processor -- start 2024-09-18T14:23:45.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 1 -- start start 2024-09-18T14:23:45.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient: init 2024-09-18T14:23:45.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T14:23:45.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f797009e830 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:45.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797009f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient(hunting): picked mon.noname-b con 0x7f797009ec00 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7970136e60 0x7f7970139240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient(hunting): picked mon.noname-a con 0x7f7970136e60 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797009b7a0 con 0x7f797009e830 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7970099f10 con 0x7f7970136e60 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f797009b4c0 con 0x7f797009ec00 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7976c3c700 10 monclient(hunting): authenticate will time out at 946.366805s 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797009f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975c3a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f797009e830 0x7f7970131d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59818/0 (socket says 172.21.15.104:59818) 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975c3a700 1 -- 172.21.15.104:0/3625842653 learned_addr learned my addr 172.21.15.104:0/3625842653 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 1 --2- 172.21.15.104:0/3625842653 >> v2:172.21.15.104:3300/0 conn(0x7f7970136e60 0x7f7970139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): get_auth_request con 0x7f797009ec00 auth_method 0 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 10 monclient(hunting): get_auth_request con 0x7f7970136e60 auth_method 0 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f797643b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient(hunting): handle_auth_done global_id 4850 payload 274 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 1 -- 172.21.15.104:0/3625842653 >> v2:172.21.15.104:3300/0 conn(0x7f7970136e60 msgr2=0x7f7970139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 1 --2- 172.21.15.104:0/3625842653 >> v2:172.21.15.104:3300/0 conn(0x7f7970136e60 0x7f7970139240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 1 -- 172.21.15.104:0/3625842653 >> v1:172.21.15.104:6789/0 conn(0x7f797009e830 legacy=0x7f7970131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 1 monclient: found mon.noname-b 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.373+0000 7f7975439700 1 -- 172.21.15.104:0/3625842653 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797009ba80 con 0x7f797009ec00 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7975439700 1 -- 172.21.15.104:0/3625842653 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 msgr2=0x7f797009f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7975439700 1 --2- 172.21.15.104:0/3625842653 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797009f000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f79680098e0 tx=0x7f79680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6b20019459ba56 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 1 -- 172.21.15.104:0/3625842653 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800b850 con 0x7f797009ec00 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: dump: 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 5 monclient: authenticate success, global_id 4850 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 1 -- 172.21.15.104:0/3625842653 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796800b9b0 con 0x7f797009ec00 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 4 set_mon_vals no callback set 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 1 -- 172.21.15.104:0/3625842653 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800bd50 con 0x7f797009ec00 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7974c38700 10 monclient: dump: 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 10 monclient: shutdown 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 -- 172.21.15.104:0/3625842653 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 msgr2=0x7f797009f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 --2- 172.21.15.104:0/3625842653 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797009f000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f79680098e0 tx=0x7f79680091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 -- 172.21.15.104:0/3625842653 shutdown_connections 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 --2- 172.21.15.104:0/3625842653 >> v2:172.21.15.104:3300/0 conn(0x7f7970136e60 0x7f7970139240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 --2- 172.21.15.104:0/3625842653 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797009f000 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 -- 172.21.15.104:0/3625842653 >> 172.21.15.104:0/3625842653 conn(0x7f79700676a0 msgr2=0x7f7970067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 -- 172.21.15.104:0/3625842653 shutdown_connections 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 1 -- 172.21.15.104:0/3625842653 wait complete. 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.377+0000 7f7976c3c700 10 monclient: monmap: 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:45.394 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 Processor -- start 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 -- start start 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient: init 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797009e830 0x7f7970130a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): picked mon.noname-b con 0x7f797009e830 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): picked mon.noname-c con 0x7f797009ec00 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7970136e60 0x7f797006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): picked mon.noname-a con 0x7f7970136e60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797011cb70 con 0x7f7970136e60 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797009f5e0 con 0x7f797009e830 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79700a4370 con 0x7f797009ec00 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7976c3c700 10 monclient(hunting): authenticate will time out at 946.373974s 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797006db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:59450/0 (socket says 172.21.15.104:59450) 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 -- 172.21.15.104:0/2059166036 learned_addr learned my addr 172.21.15.104:0/2059166036 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): get_auth_request con 0x7f797009ec00 auth_method 0 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7970136e60 0x7f797006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797009e830 0x7f7970130a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 10 monclient(hunting): get_auth_request con 0x7f7970136e60 auth_method 0 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 10 monclient(hunting): get_auth_request con 0x7f797009e830 auth_method 0 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f797643b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient(hunting): handle_auth_done global_id 4853 payload 824 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797009e830 msgr2=0x7f7970130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797009e830 0x7f7970130a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7970136e60 msgr2=0x7f797006e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7970136e60 0x7f797006e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 monclient: found mon.noname-c 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.381+0000 7f7975439700 1 -- 172.21.15.104:0/2059166036 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7968009420 con 0x7f797009ec00 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975439700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 msgr2=0x7f797006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975439700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797006db70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79680030c0 tx=0x7f7968016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968000c80 con 0x7f797009ec00 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: dump: 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796801b470 con 0x7f797009ec00 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796801b830 con 0x7f797009ec00 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 10 monclient: dump: 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 5 monclient: authenticate success, global_id 4853 2024-09-18T14:23:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 10 monclient: _renew_subs 2024-09-18T14:23:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797006fae0 con 0x7f797009ec00 2024-09-18T14:23:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796ffff700 4 set_mon_vals no callback set 2024-09-18T14:23:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 10 monclient: _renew_subs 2024-09-18T14:23:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:23:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797006f5d0 con 0x7f797009ec00 2024-09-18T14:23:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 0x7f7970029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7970064840 con 0x7f7970130fa0 2024-09-18T14:23:45.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f796801bba0 con 0x7f797009ec00 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 0x7f7970029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 10 monclient: get_auth_request con 0x7f7970130fa0 auth_method 0 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 1 --2- 172.21.15.104:0/2059166036 >> v2:172.21.15.174:6812/17603 conn(0x7f795c035ad0 0x7f795c037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.385+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7968049920 con 0x7f797009ec00 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f7975c3a700 10 monclient(hunting): handle_auth_done global_id 4873 payload 824 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f7975c3a700 1 -- 172.21.15.104:0/2059166036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79701363a0 con 0x7f7970130fa0 2024-09-18T14:23:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f797643b700 1 --2- 172.21.15.104:0/2059166036 >> v2:172.21.15.174:6812/17603 conn(0x7f795c035ad0 0x7f795c037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f797643b700 10 monclient: get_auth_request con 0x7f795c035ad0 auth_method 0 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f7975c3a700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 msgr2=0x7f7970029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f7975c3a700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 0x7f7970029d00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f796400eac0 tx=0x7f7964007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400cc40 con 0x7f7970130fa0 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 10 monclient: dump: 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f797643b700 1 --2- 172.21.15.104:0/2059166036 >> v2:172.21.15.174:6812/17603 conn(0x7f795c035ad0 0x7f795c037f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7960009fd0 tx=0x7f796000b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f796400cda0 con 0x7f7970130fa0 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 msgr2=0x7f7970029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 0x7f7970029d00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f796400eac0 tx=0x7f7964007f90 comp rx=0 tx=0).stop 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 msgr2=0x7f7970029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.389+0000 7f796dffb700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7970130fa0 0x7f7970029d00 secure :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f796400eac0 tx=0x7f7964007f90 comp rx=0 tx=0).stop 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7976c3c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7976c3c700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 0x7f797002a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7976c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7970131370 con 0x7f797011cd90 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 0x7f797002a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient: get_auth_request con 0x7f797011cd90 auth_method 0 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 10 monclient(hunting): handle_auth_done global_id 4876 payload 824 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.401+0000 7f7975c3a700 1 -- 172.21.15.104:0/2059166036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7970063450 con 0x7f797011cd90 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7975c3a700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 msgr2=0x7f797002a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7975c3a700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 0x7f797002a240 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f79640070c0 tx=0x7f7964016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964013d10 con 0x7f797011cd90 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 10 monclient: dump: 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.411 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f7964013e70 con 0x7f797011cd90 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 msgr2=0x7f797002a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 0x7f797002a240 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f79640070c0 tx=0x7f7964016000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 msgr2=0x7f797002a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f796dffb700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f797011cd90 0x7f797002a240 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f79640070c0 tx=0x7f7964016000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.412 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 20, 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 5, 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:45.413 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:23:45.414 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.415 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:23:45.416 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:23:45.417 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.419 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.420 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:23:45.421 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:23:45.422 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.423 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:23:45.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 7 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.425 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:23:45.426 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 >> v2:172.21.15.174:6812/17603 conn(0x7f795c035ad0 msgr2=0x7f795c037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 --2- 172.21.15.104:0/2059166036 >> v2:172.21.15.174:6812/17603 conn(0x7f795c035ad0 0x7f795c037f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7960009fd0 tx=0x7f796000b040 comp rx=0 tx=0).stop 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 10 monclient: shutdown 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 msgr2=0x7f797006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 --2- 172.21.15.104:0/2059166036 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f797009ec00 0x7f797006db70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79680030c0 tx=0x7f7968016000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f797643b700 1 -- 172.21.15.104:0/2059166036 reap_dead start 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f797643b700 1 -- 172.21.15.104:0/2059166036 reap_dead start 2024-09-18T14:23:45.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 shutdown_connections 2024-09-18T14:23:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 >> 172.21.15.104:0/2059166036 conn(0x7f79700676a0 msgr2=0x7f797009d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 shutdown_connections 2024-09-18T14:23:45.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.405+0000 7f7976c3c700 1 -- 172.21.15.104:0/2059166036 wait complete. 2024-09-18T14:23:45.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:23:45.429 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T14:23:45.429 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T14:23:45.429 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T14:23:45.429 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T14:23:45.430 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T14:23:45.430 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:23:45.446 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T14:23:45.446 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T14:23:45.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient: get_monmap_and_config 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient: monmap: 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:45.599878+0000 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:45.599878+0000 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:45.601 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 1 Processor -- start 2024-09-18T14:23:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 1 -- start start 2024-09-18T14:23:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient: init 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e006e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient(hunting): picked mon.noname-b con 0x7f87e006e230 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87e006d720 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f87e009c940 0x7f87e009cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient(hunting): picked mon.noname-a con 0x7f87e009c940 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e009e040 con 0x7f87e006d720 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87de59c700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 msgr2=0x7f87e006e600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e006cb30 con 0x7f87e009c940 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87de59c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e006e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e006c850 con 0x7f87e006e230 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87df59e700 10 monclient(hunting): authenticate will time out at 946.591325s 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f87e009c940 0x7f87e009cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ddd9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f87e006d720 0x7f87e009c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59820/0 (socket says 172.21.15.104:59820) 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ddd9b700 1 -- 172.21.15.104:0/4141583813 learned_addr learned my addr 172.21.15.104:0/4141583813 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): get_auth_request con 0x7f87e009c940 auth_method 0 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient(hunting): handle_auth_done global_id 14184 payload 274 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 1 -- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 msgr2=0x7f87e006e600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 1 --2- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e006e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 1 -- 172.21.15.104:0/4141583813 >> v1:172.21.15.104:6789/0 conn(0x7f87e006d720 legacy=0x7f87e009c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 1 monclient: found mon.noname-a 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.597+0000 7f87ded9d700 1 -- 172.21.15.104:0/4141583813 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e006c570 con 0x7f87e009c940 2024-09-18T14:23:45.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87ded9d700 1 -- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 msgr2=0x7f87e009cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87ded9d700 1 --2- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e009cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009e60 tx=0x7f87d8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30613cdbbe79ce8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 1 -- 172.21.15.104:0/4141583813 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d800e070 con 0x7f87e009c940 2024-09-18T14:23:45.608 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: dump: 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 1 -- 172.21.15.104:0/4141583813 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8012430 con 0x7f87e009c940 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 5 monclient: authenticate success, global_id 14184 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 1 -- 172.21.15.104:0/4141583813 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d80127d0 con 0x7f87e009c940 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 4 set_mon_vals no callback set 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87dd59a700 10 monclient: dump: 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.610 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 10 monclient: shutdown 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 1 -- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 msgr2=0x7f87e009cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 1 --2- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e009cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009e60 tx=0x7f87d8009780 comp rx=0 tx=0).stop 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 1 -- 172.21.15.104:0/4141583813 shutdown_connections 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 1 --2- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e009cd40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 1 --2- 172.21.15.104:0/4141583813 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e006e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.601+0000 7f87df59e700 1 -- 172.21.15.104:0/4141583813 >> 172.21.15.104:0/4141583813 conn(0x7f87e0068120 msgr2=0x7f87e00a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 -- 172.21.15.104:0/4141583813 shutdown_connections 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 -- 172.21.15.104:0/4141583813 wait complete. 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient: build_initial_monmap 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient: monmap: 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:45.612 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 Processor -- start 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 -- start start 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient: init 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87e006d720 0x7f87e013f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): picked mon.noname-b con 0x7f87e006d720 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e013fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): picked mon.noname-c con 0x7f87e006e230 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): picked mon.noname-a con 0x7f87e009c940 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0118550 con 0x7f87e009c940 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e009bce0 con 0x7f87e006d720 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e009b230 con 0x7f87e006e230 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87df59e700 10 monclient(hunting): authenticate will time out at 946.599359s 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ddd9b700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 msgr2=0x7f87e013fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ddd9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e013fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e0133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53634/0 (socket says 172.21.15.104:53634) 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 1 -- 172.21.15.104:0/852933163 learned_addr learned my addr 172.21.15.104:0/852933163 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 10 monclient(hunting): get_auth_request con 0x7f87e009c940 auth_method 0 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87e006d720 0x7f87e013f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 10 monclient(hunting): get_auth_request con 0x7f87e006d720 auth_method 0 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87ded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.605+0000 7f87de59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 10 monclient(hunting): handle_auth_done global_id 14187 payload 824 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 -- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 msgr2=0x7f87e013fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:23:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e013fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 -- 172.21.15.104:0/852933163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87e006d720 msgr2=0x7f87e013f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87e006d720 0x7f87e013f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 10 monclient: _finish_hunting 0 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 monclient: found mon.noname-a 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 -- 172.21.15.104:0/852933163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d8009a20 con 0x7f87e009c940 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 -- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 msgr2=0x7f87e0133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ded9d700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e0133b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f87d800b6c0 tx=0x7f87d8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 1 -- 172.21.15.104:0/852933163 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8026430 con 0x7f87e009c940 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: dump: 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87df59e700 5 monclient: authenticate success, global_id 14187 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 1 -- 172.21.15.104:0/852933163 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8021070 con 0x7f87e009c940 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 1 -- 172.21.15.104:0/852933163 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8026820 con 0x7f87e009c940 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87df59e700 10 monclient: _renew_subs 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87df59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e01187a0 con 0x7f87e009c940 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87ce7fc700 10 monclient: dump: 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:45.622 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87dcd99700 4 set_mon_vals no callback set 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87df59e700 10 monclient: _renew_subs 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87df59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.609+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e0135740 con 0x7f87e009c940 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87ce7fc700 1 -- 172.21.15.104:0/852933163 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f87d8023030 con 0x7f87e009c940 2024-09-18T14:23:45.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87ce7fc700 1 --2- 172.21.15.104:0/852933163 >> v2:172.21.15.174:6812/17603 conn(0x7f87c0035a30 0x7f87c0037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87ce7fc700 1 -- 172.21.15.104:0/852933163 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f87d804a020 con 0x7f87e009c940 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87df59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87df59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87df59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87de59c700 1 --2- 172.21.15.104:0/852933163 >> v2:172.21.15.174:6812/17603 conn(0x7f87c0035a30 0x7f87c0037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e009e040 con 0x7f87e009c940 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.613+0000 7f87de59c700 10 monclient: get_auth_request con 0x7f87c0035a30 auth_method 0 2024-09-18T14:23:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.617+0000 7f87de59c700 1 --2- 172.21.15.104:0/852933163 >> v2:172.21.15.174:6812/17603 conn(0x7f87c0035a30 0x7f87c0037ee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f87d00039d0 tx=0x7f87d00091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:45.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.621+0000 7f87ce7fc700 1 -- 172.21.15.104:0/852933163 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f87e009e040 con 0x7f87e009c940 2024-09-18T14:23:45.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.621+0000 7f87ce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:45.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.621+0000 7f87ce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:45.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-18T14:23:45.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.849+0000 7f87df59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.849+0000 7f87df59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.849+0000 7f87df59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.849+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f87e009e040 con 0x7f87e009c940 2024-09-18T14:23:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.849+0000 7f87ce7fc700 1 -- 172.21.15.104:0/852933163 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f87e009e040 con 0x7f87e009c940 2024-09-18T14:23:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.849+0000 7f87ce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.849+0000 7f87ce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:45.855 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 >> v2:172.21.15.174:6812/17603 conn(0x7f87c0035a30 msgr2=0x7f87c0037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 --2- 172.21.15.104:0/852933163 >> v2:172.21.15.174:6812/17603 conn(0x7f87c0035a30 0x7f87c0037ee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f87d00039d0 tx=0x7f87d00091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 10 monclient: shutdown 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 msgr2=0x7f87e0133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e0133b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f87d800b6c0 tx=0x7f87d8018000 comp rx=0 tx=0).stop 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 shutdown_connections 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 --2- 172.21.15.104:0/852933163 >> v2:172.21.15.174:6812/17603 conn(0x7f87c0035a30 0x7f87c0037ee0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87e009c940 0x7f87e0133b80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f87e006e230 0x7f87e013fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 --2- 172.21.15.104:0/852933163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87e006d720 0x7f87e013f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 >> 172.21.15.104:0/852933163 conn(0x7f87e0068120 msgr2=0x7f87e009a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 shutdown_connections 2024-09-18T14:23:45.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:45.853+0000 7f87df59e700 1 -- 172.21.15.104:0/852933163 wait complete. 2024-09-18T14:23:48.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:48.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.981+0000 7f3c4b59e700 10 monclient: get_monmap_and_config 2024-09-18T14:23:48.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.981+0000 7f3c4b59e700 10 monclient: build_initial_monmap 2024-09-18T14:23:48.987 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.981+0000 7f3c4b59e700 10 monclient: monmap: 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:48.987594+0000 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:48.987594+0000 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:48.988 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 1 Processor -- start 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 1 -- start start 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient: init 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c09a5b0 0x7f3c4c09ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f3c4c09a5b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:48.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3c4c12f980 0x7f3c4c12fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f3c4c12f980 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c4c130290 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4c097d10 con 0x7f3c4c130290 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4c098470 con 0x7f3c4c12f980 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c099650 con 0x7f3c4c09a5b0 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4b59e700 10 monclient(hunting): authenticate will time out at 949.977207s 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4a59c700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c09a5b0 msgr2=0x7f3c4c09ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4a59c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c09a5b0 0x7f3c4c09ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4ad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3c4c130290 0x7f3c4c12c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59826/0 (socket says 172.21.15.104:59826) 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c4ad9d700 1 -- 172.21.15.104:0/1411358642 learned_addr learned my addr 172.21.15.104:0/1411358642 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 1 --2- 172.21.15.104:0/1411358642 >> v2:172.21.15.174:3300/0 conn(0x7f3c4c12f980 0x7f3c4c12fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): get_auth_request con 0x7f3c4c12f980 auth_method 0 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient(hunting): handle_auth_done global_id 4882 payload 274 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 1 -- 172.21.15.104:0/1411358642 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c09a5b0 msgr2=0x7f3c4c09ae20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 1 --2- 172.21.15.104:0/1411358642 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c09a5b0 0x7f3c4c09ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 1 -- 172.21.15.104:0/1411358642 >> v1:172.21.15.104:6789/0 conn(0x7f3c4c130290 legacy=0x7f3c4c12c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient: _finish_hunting 0 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 1 monclient: found mon.noname-c 2024-09-18T14:23:48.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:48.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:23:48.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.985+0000 7f3c49d9b700 1 -- 172.21.15.104:0/1411358642 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c099930 con 0x7f3c4c12f980 2024-09-18T14:23:48.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c49d9b700 1 -- 172.21.15.104:0/1411358642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c12f980 msgr2=0x7f3c4c12fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:48.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c49d9b700 1 --2- 172.21.15.104:0/1411358642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12fd50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009e60 tx=0x7f3c44009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1e501a0fc0d1f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:48.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 1 -- 172.21.15.104:0/1411358642 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4400e070 con 0x7f3c4c12f980 2024-09-18T14:23:48.994 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: dump: 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: _finish_auth 0 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 5 monclient: authenticate success, global_id 4882 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 1 -- 172.21.15.104:0/1411358642 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c44012430 con 0x7f3c4c12f980 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 1 -- 172.21.15.104:0/1411358642 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440127d0 con 0x7f3c4c12f980 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4959a700 10 monclient: dump: 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 4 set_mon_vals no callback set 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 10 monclient: shutdown 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 -- 172.21.15.104:0/1411358642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c12f980 msgr2=0x7f3c4c12fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/1411358642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12fd50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009e60 tx=0x7f3c44009780 comp rx=0 tx=0).stop 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 -- 172.21.15.104:0/1411358642 shutdown_connections 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/1411358642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12fd50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/1411358642 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c09a5b0 0x7f3c4c09ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 -- 172.21.15.104:0/1411358642 >> 172.21.15.104:0/1411358642 conn(0x7f3c4c0a12c0 msgr2=0x7f3c4c0a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 -- 172.21.15.104:0/1411358642 shutdown_connections 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 -- 172.21.15.104:0/1411358642 wait complete. 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 10 monclient: build_initial_monmap 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 10 monclient: monmap: 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:48.999 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:48.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 Processor -- start 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 1 -- start start 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.989+0000 7f3c4b59e700 10 monclient: init 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c09a5b0 0x7f3c4c06edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f3c4c09a5b0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f3c4c12f980 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c130290 0x7f3c4c117580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f3c4c130290 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c1188c0 con 0x7f3c4c12f980 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0989a0 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c132a80 con 0x7f3c4c130290 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4b59e700 10 monclient(hunting): authenticate will time out at 949.984549s 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4ad9d700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c130290 msgr2=0x7f3c4c117580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4ad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c130290 0x7f3c4c117580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53646/0 (socket says 172.21.15.104:53646) 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 1 -- 172.21.15.104:0/3576739743 learned_addr learned my addr 172.21.15.104:0/3576739743 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 10 monclient(hunting): get_auth_request con 0x7f3c4c12f980 auth_method 0 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c09a5b0 0x7f3c4c06edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): get_auth_request con 0x7f3c4c09a5b0 auth_method 0 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient(hunting): handle_auth_done global_id 4885 payload 824 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 1 -- 172.21.15.104:0/3576739743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c130290 msgr2=0x7f3c4c117580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c130290 0x7f3c4c117580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 1 -- 172.21.15.104:0/3576739743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c4c12f980 msgr2=0x7f3c4c12ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient: _finish_hunting 0 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 1 monclient: found mon.noname-b 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.174:3300/0 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c4a59c700 1 -- 172.21.15.104:0/3576739743 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c44009a20 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.993+0000 7f3c49d9b700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:23:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4a59c700 1 -- 172.21.15.104:0/3576739743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c09a5b0 msgr2=0x7f3c4c06edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4a59c700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c09a5b0 0x7f3c4c06edc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400b600 tx=0x7f3c340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 1 -- 172.21.15.104:0/3576739743 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34003e20 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: dump: 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: _finish_auth 0 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 1 -- 172.21.15.104:0/3576739743 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c34006150 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 5 monclient: authenticate success, global_id 4885 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 10 monclient: _renew_subs 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4c119b80 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 1 -- 172.21.15.104:0/3576739743 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34006510 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:49.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 10 monclient: dump: 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c48d99700 4 set_mon_vals no callback set 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 10 monclient: _renew_subs 2024-09-18T14:23:49.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4c13fd10 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 1 -- 172.21.15.104:0/3576739743 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f3c3401b3b0 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c11a0a0 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 1 --2- 172.21.15.104:0/3576739743 >> v2:172.21.15.174:6812/17603 conn(0x7f3c2c035a30 0x7f3c2c037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c427fc700 1 -- 172.21.15.104:0/3576739743 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3c34047410 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c49d9b700 1 --2- 172.21.15.104:0/3576739743 >> v2:172.21.15.174:6812/17603 conn(0x7f3c2c035a30 0x7f3c2c037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:48.997+0000 7f3c49d9b700 10 monclient: get_auth_request con 0x7f3c2c035a30 auth_method 0 2024-09-18T14:23:49.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.001+0000 7f3c49d9b700 1 --2- 172.21.15.104:0/3576739743 >> v2:172.21.15.174:6812/17603 conn(0x7f3c2c035a30 0x7f3c2c037ee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009870 tx=0x7f3c44018000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:49.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.009+0000 7f3c427fc700 1 -- 172.21.15.104:0/3576739743 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3c3400e680 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.013+0000 7f3c427fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:49.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.013+0000 7f3c427fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:49.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.237+0000 7f3c4b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:49.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.237+0000 7f3c4b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:49.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.237+0000 7f3c4b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:23:49.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.237+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3c4c0633b0 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.237+0000 7f3c427fc700 1 -- 172.21.15.104:0/3576739743 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f3c34017030 con 0x7f3c4c09a5b0 2024-09-18T14:23:49.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.237+0000 7f3c427fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:49.242 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.237+0000 7f3c427fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:49.242 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:49.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 >> v2:172.21.15.174:6812/17603 conn(0x7f3c2c035a30 msgr2=0x7f3c2c037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/3576739743 >> v2:172.21.15.174:6812/17603 conn(0x7f3c2c035a30 0x7f3c2c037ee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009870 tx=0x7f3c44018000 comp rx=0 tx=0).stop 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 10 monclient: shutdown 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c09a5b0 msgr2=0x7f3c4c06edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c09a5b0 0x7f3c4c06edc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400b600 tx=0x7f3c340091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 shutdown_connections 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/3576739743 >> v2:172.21.15.174:6812/17603 conn(0x7f3c2c035a30 0x7f3c2c037ee0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3c4c130290 0x7f3c4c117580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c4c12f980 0x7f3c4c12ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 --2- 172.21.15.104:0/3576739743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c4c09a5b0 0x7f3c4c06edc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 >> 172.21.15.104:0/3576739743 conn(0x7f3c4c0a12c0 msgr2=0x7f3c4c132c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:49.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 shutdown_connections 2024-09-18T14:23:49.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:49.241+0000 7f3c4b59e700 1 -- 172.21.15.104:0/3576739743 wait complete. 2024-09-18T14:23:51.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-18T14:23:52.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:52.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.373+0000 7fa62d3ca700 10 monclient: get_monmap_and_config 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.373+0000 7fa62d3ca700 10 monclient: build_initial_monmap 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.373+0000 7fa62d3ca700 10 monclient: monmap: 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:52.379167+0000 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:52.379167+0000 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:52.381 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:52.381 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:52.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 1 Processor -- start 2024-09-18T14:23:52.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 1 -- start start 2024-09-18T14:23:52.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient: init 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa6280a59d0 0x7fa62809ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient(hunting): picked mon.noname-a con 0x7fa6280a59d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa62809bea0 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa62809f240 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62809af10 con 0x7fa62809f240 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6280a1870 con 0x7fa62809bea0 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:52.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6280a5da0 con 0x7fa6280a59d0 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62d3ca700 10 monclient(hunting): authenticate will time out at 953.368735s 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62cbc9700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa62809f240 0x7fa6280a03c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57364/0 (socket says 172.21.15.104:57364) 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa62cbc9700 1 -- 172.21.15.104:0/263610322 learned_addr learned my addr 172.21.15.104:0/263610322 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 1 --2- 172.21.15.104:0/263610322 >> v2:172.21.15.104:3300/0 conn(0x7fa6280a59d0 0x7fa62809ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): get_auth_request con 0x7fa6280a59d0 auth_method 0 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:52.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient(hunting): handle_auth_done global_id 14193 payload 274 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 1 -- 172.21.15.104:0/263610322 >> v1:172.21.15.174:6789/0 conn(0x7fa62809bea0 legacy=0x7fa62809fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 1 -- 172.21.15.104:0/263610322 >> v1:172.21.15.104:6789/0 conn(0x7fa62809f240 legacy=0x7fa6280a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient: _finish_hunting 0 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 1 monclient: found mon.noname-a 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.377+0000 7fa627fff700 1 -- 172.21.15.104:0/263610322 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6280a0ad0 con 0x7fa6280a59d0 2024-09-18T14:23:52.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa627fff700 1 -- 172.21.15.104:0/263610322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6280a59d0 msgr2=0x7fa62809ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:52.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa627fff700 1 --2- 172.21.15.104:0/263610322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6280a59d0 0x7fa62809ed00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa62806f350 tx=0x7fa6180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f40da18d85a89ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:52.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 1 -- 172.21.15.104:0/263610322 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61800b9a0 con 0x7fa6280a59d0 2024-09-18T14:23:52.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:52.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: dump: 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:52.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: _finish_auth 0 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 5 monclient: authenticate success, global_id 14193 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 1 -- 172.21.15.104:0/263610322 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61800bb00 con 0x7fa6280a59d0 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient: get_monmap_and_config success 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 4 set_mon_vals no callback set 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 1 -- 172.21.15.104:0/263610322 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61800bea0 con 0x7fa6280a59d0 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa6277fe700 10 monclient: dump: 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:52.393 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient: shutdown 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- 172.21.15.104:0/263610322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6280a59d0 msgr2=0x7fa62809ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/263610322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6280a59d0 0x7fa62809ed00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa62806f350 tx=0x7fa6180091a0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.394 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- 172.21.15.104:0/263610322 shutdown_connections 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/263610322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6280a59d0 0x7fa62809ed00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- 172.21.15.104:0/263610322 >> 172.21.15.104:0/263610322 conn(0x7fa628067c10 msgr2=0x7fa6280a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- 172.21.15.104:0/263610322 shutdown_connections 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- 172.21.15.104:0/263610322 wait complete. 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient: build_initial_monmap 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient: monmap: 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:52.395 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 Processor -- start 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- start start 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient: init 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa62809bea0 0x7fa6281412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): picked mon.noname-c con 0x7fa62809bea0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 0x7fa628131cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): picked mon.noname-a con 0x7fa62809f240 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:23:52.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6280a59d0 0x7fa62811b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): picked mon.noname-b con 0x7fa6280a59d0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62809e400 con 0x7fa62809f240 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62811f0f0 con 0x7fa6280a59d0 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62811ee10 con 0x7fa62809bea0 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62d3ca700 10 monclient(hunting): authenticate will time out at 953.375028s 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa627fff700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa62809bea0 msgr2=0x7fa6281412b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa627fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa62809bea0 0x7fa6281412b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 0x7fa628131cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:52.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 0x7fa628131cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60624/0 (socket says 172.21.15.104:60624) 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 -- 172.21.15.104:0/2050130964 learned_addr learned my addr 172.21.15.104:0/2050130964 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): get_auth_request con 0x7fa62809f240 auth_method 0 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62cbc9700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6280a59d0 0x7fa62811b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient(hunting): handle_auth_done global_id 14196 payload 824 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 -- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa62809bea0 msgr2=0x7fa6281412b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa62809bea0 0x7fa6281412b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 -- 172.21.15.104:0/2050130964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6280a59d0 msgr2=0x7fa62811b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6280a59d0 0x7fa62811b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient: _finish_hunting 0 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 monclient: found mon.noname-a 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa61ffff700 1 -- 172.21.15.104:0/2050130964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618009420 con 0x7fa62809f240 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62cbc9700 10 monclient: get_auth_request con 0x7fa6280a59d0 auth_method 0 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.381+0000 7fa62cbc9700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6280a59d0 0x7fa62811b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa61ffff700 1 -- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 msgr2=0x7fa628131cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa61ffff700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 0x7fa628131cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b5e0 tx=0x7fa610009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 1 -- 172.21.15.104:0/2050130964 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000f800 con 0x7fa62809f240 2024-09-18T14:23:52.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: dump: 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:52.400 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: _finish_auth 0 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa62d3ca700 5 monclient: authenticate success, global_id 14196 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 1 -- 172.21.15.104:0/2050130964 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61000f980 con 0x7fa62809f240 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa62d3ca700 10 monclient: _renew_subs 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa62d3ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62811f3d0 con 0x7fa62809f240 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 1 -- 172.21.15.104:0/2050130964 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000fd40 con 0x7fa62809f240 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa626ffd700 4 set_mon_vals no callback set 2024-09-18T14:23:52.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 10 monclient: dump: 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa62d3ca700 10 monclient: _renew_subs 2024-09-18T14:23:52.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa62d3ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:52.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62811e180 con 0x7fa62809f240 2024-09-18T14:23:52.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 1 -- 172.21.15.104:0/2050130964 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fa610011610 con 0x7fa62809f240 2024-09-18T14:23:52.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.385+0000 7fa624ff9700 1 --2- 172.21.15.104:0/2050130964 >> v2:172.21.15.174:6812/17603 conn(0x7fa608035a30 0x7fa608037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:52.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa624ff9700 1 -- 172.21.15.104:0/2050130964 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa610047b30 con 0x7fa62809f240 2024-09-18T14:23:52.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa627fff700 1 --2- 172.21.15.104:0/2050130964 >> v2:172.21.15.174:6812/17603 conn(0x7fa608035a30 0x7fa608037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:52.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa627fff700 10 monclient: get_auth_request con 0x7fa608035a30 auth_method 0 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa62d3ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa62d3ca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa62d3ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6280633b0 con 0x7fa62809f240 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.389+0000 7fa627fff700 1 --2- 172.21.15.104:0/2050130964 >> v2:172.21.15.174:6812/17603 conn(0x7fa608035a30 0x7fa608037ee0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6180093a0 tx=0x7fa618016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.393+0000 7fa624ff9700 1 -- 172.21.15.104:0/2050130964 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa61001a400 con 0x7fa62809f240 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.393+0000 7fa624ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:23:52.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.393+0000 7fa624ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:23:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.625+0000 7fa62d3ca700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:23:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.625+0000 7fa62d3ca700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.625+0000 7fa62d3ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:23:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.625+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa628068950 con 0x7fa62809f240 2024-09-18T14:23:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.625+0000 7fa624ff9700 1 -- 172.21.15.104:0/2050130964 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa6100172b0 con 0x7fa62809f240 2024-09-18T14:23:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.625+0000 7fa624ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:23:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.625+0000 7fa624ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:23:52.630 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:23:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 >> v2:172.21.15.174:6812/17603 conn(0x7fa608035a30 msgr2=0x7fa608037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/2050130964 >> v2:172.21.15.174:6812/17603 conn(0x7fa608035a30 0x7fa608037ee0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6180093a0 tx=0x7fa618016000 comp rx=0 tx=0).stop 2024-09-18T14:23:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 10 monclient: shutdown 2024-09-18T14:23:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:23:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 msgr2=0x7fa628131cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 0x7fa628131cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b5e0 tx=0x7fa610009120 comp rx=0 tx=0).stop 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 shutdown_connections 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/2050130964 >> v2:172.21.15.174:6812/17603 conn(0x7fa608035a30 0x7fa608037ee0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6280a59d0 0x7fa62811b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa62809f240 0x7fa628131cf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 --2- 172.21.15.104:0/2050130964 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa62809bea0 0x7fa6281412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 >> 172.21.15.104:0/2050130964 conn(0x7fa628067c10 msgr2=0x7fa62809cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 shutdown_connections 2024-09-18T14:23:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:52.629+0000 7fa62d3ca700 1 -- 172.21.15.104:0/2050130964 wait complete. 2024-09-18T14:23:55.644 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:23:55.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.753+0000 7f67f6384700 10 monclient: get_monmap_and_config 2024-09-18T14:23:55.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.753+0000 7f67f6384700 10 monclient: build_initial_monmap 2024-09-18T14:23:55.759 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.753+0000 7f67f6384700 10 monclient: monmap: 2024-09-18T14:23:55.759 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:23:55.759533+0000 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:23:55.759533+0000 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:23:55.760 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:23:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 1 Processor -- start 2024-09-18T14:23:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 1 -- start start 2024-09-18T14:23:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient: init 2024-09-18T14:23:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient: _reopen_session rank -1 2024-09-18T14:23:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T14:23:55.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67f012de40 addr v1:172.21.15.174:6789/0 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient(hunting): picked mon.noname-c con 0x7f67f006a110 addr v2:172.21.15.174:3300/0 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67f009ad70 addr v1:172.21.15.104:6789/0 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f0095ee0 con 0x7f67f009ad70 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f0097570 con 0x7f67f012de40 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f0097850 con 0x7f67f006a110 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f6384700 10 monclient(hunting): authenticate will time out at 956.749268s 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f5b83700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f67f009ad70 0x7f67f01360f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57366/0 (socket says 172.21.15.104:57366) 2024-09-18T14:23:55.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f5b83700 1 -- 172.21.15.104:0/3332278309 learned_addr learned my addr 172.21.15.104:0/3332278309 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67f006a110 auth_method 0 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:55.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.757+0000 7f67f5b83700 1 -- 172.21.15.104:0/3332278309 >> v1:172.21.15.104:6789/0 conn(0x7f67f009ad70 legacy=0x7f67f01360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:55.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.761+0000 7f67f5382700 1 -- 172.21.15.104:0/3332278309 >> v1:172.21.15.174:6789/0 conn(0x7f67f012de40 legacy=0x7f67f009a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:23:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67f006a110 auth_method 0 2024-09-18T14:23:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:23:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:55.961+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:23:56.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:56.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67f006a110 auth_method 0 2024-09-18T14:23:56.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:56.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:56.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:56.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:23:56.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:56.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:56.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:56.365+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:23:57.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:57.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67f006a110 auth_method 0 2024-09-18T14:23:57.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:57.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:57.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:57.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:23:57.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:57.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:57.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:57.169+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient(hunting): tick 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 monclient(hunting): continuing hunt 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 msgr2=0x7f67f012e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.174:3300/0 conn(0x7f67f006a110 0x7f67f012e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 -- 172.21.15.104:0/3332278309 >> v1:172.21.15.174:6789/0 conn(0x7f67f012de40 legacy=0x7f67f009a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 -- 172.21.15.104:0/3332278309 >> v1:172.21.15.104:6789/0 conn(0x7f67f009ad70 legacy=0x7f67f01360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient: _add_conns ranks=[2,3] 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 0x7f67d4005ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:58.762 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient(hunting): picked mon.noname-b con 0x7f67d4003880 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f67d40063d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 -- 172.21.15.104:0/3332278309 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f0097850 con 0x7f67d40063d0 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 1 -- 172.21.15.104:0/3332278309 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d4008f40 con 0x7f67d4003880 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67e6ffd700 10 monclient(hunting): _renew_subs 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f5382700 1 -- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 msgr2=0x7f67d4005ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f5382700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 0x7f67d4005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67d40063d0 auth_method 0 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:58.763 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:58.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:23:58.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 msgr2=0x7f67d4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:58.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 msgr2=0x7f67d4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:58.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:58.764 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.757+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:23:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f5382700 1 -- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 msgr2=0x7f67d4005ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f5382700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 0x7f67d4005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:23:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67d40063d0 auth_method 0 2024-09-18T14:23:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 msgr2=0x7f67d4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:23:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 msgr2=0x7f67d4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:58.961+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f5382700 1 -- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 msgr2=0x7f67d4005ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f5382700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 0x7f67d4005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67d40063d0 auth_method 0 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:23:59.368 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 msgr2=0x7f67d4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:23:59.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 msgr2=0x7f67d4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:23:59.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:23:59.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:23:59.365+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:24:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f5382700 1 -- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 msgr2=0x7f67d4005ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:24:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f5382700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 0x7f67d4005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:24:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> v2:172.21.15.104:3300/0 conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67d40063d0 auth_method 0 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient(hunting): handle_auth_done global_id 14199 payload 274 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 msgr2=0x7f67d4005ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d4003880 0x7f67d4005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient: _finish_hunting 0 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 1 monclient: found mon.noname-a 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:24:00.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f009c050 con 0x7f67d40063d0 2024-09-18T14:24:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 1 -- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d40063d0 msgr2=0x7f67d4008880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.169+0000 7f67f4b81700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d40063d0 0x7f67d4008880 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67f006a530 tx=0x7f67e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757b5589926dd6ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 1 -- 172.21.15.104:0/3332278309 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e801a440 con 0x7f67d40063d0 2024-09-18T14:24:00.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: dump: 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: _finish_auth 0 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 5 monclient: authenticate success, global_id 14199 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 1 -- 172.21.15.104:0/3332278309 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e801e070 con 0x7f67d40063d0 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 1 -- 172.21.15.104:0/3332278309 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e801a7b0 con 0x7f67d40063d0 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67e7fff700 10 monclient: dump: 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.177 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.178 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.178 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.178 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.178 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 10 monclient: get_monmap_and_config success 2024-09-18T14:24:00.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 4 set_mon_vals no callback set 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 10 monclient: shutdown 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 1 -- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d40063d0 msgr2=0x7f67d4008880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d40063d0 0x7f67d4008880 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67f006a530 tx=0x7f67e800b040 comp rx=0 tx=0).stop 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f5b83700 1 -- 172.21.15.104:0/3332278309 reap_dead start 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 1 -- 172.21.15.104:0/3332278309 shutdown_connections 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 1 --2- 172.21.15.104:0/3332278309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67d40063d0 0x7f67d4008880 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 1 -- 172.21.15.104:0/3332278309 >> 172.21.15.104:0/3332278309 conn(0x7f67f00a12c0 msgr2=0x7f67f00a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:24:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 1 -- 172.21.15.104:0/3332278309 shutdown_connections 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 1 -- 172.21.15.104:0/3332278309 wait complete. 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 10 monclient: build_initial_monmap 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.173+0000 7f67f6384700 10 monclient: monmap: 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:24:00.180 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 Processor -- start 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 -- start start 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient: init 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient: _reopen_session rank -1 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d40063d0 0x7f67f006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): picked mon.noname-c con 0x7f67d40063d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67f012fb10 0x7f67f006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): picked mon.noname-b con 0x7f67f012fb10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 0x7f67f006f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): picked mon.noname-a con 0x7f67f006f0a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f012efd0 con 0x7f67f006f0a0 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f0137430 con 0x7f67f012fb10 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f006fa40 con 0x7f67d40063d0 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5382700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d40063d0 msgr2=0x7f67f006e620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): _renew_subs 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f6384700 10 monclient(hunting): authenticate will time out at 961.170161s 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5382700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d40063d0 0x7f67f006e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 0x7f67f006f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 0x7f67f006f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51304/0 (socket says 172.21.15.104:51304) 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 -- 172.21.15.104:0/581158286 learned_addr learned my addr 172.21.15.104:0/581158286 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:24:00.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): get_auth_request con 0x7f67f006f0a0 auth_method 0 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67f012fb10 0x7f67f006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 10 monclient(hunting): get_auth_request con 0x7f67f012fb10 auth_method 0 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f4b81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient(hunting): handle_auth_done global_id 14202 payload 824 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 -- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d40063d0 msgr2=0x7f67f006e620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d40063d0 0x7f67f006e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 -- 172.21.15.104:0/581158286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67f012fb10 msgr2=0x7f67f006eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67f012fb10 0x7f67f006eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient: _finish_hunting 0 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 monclient: found mon.noname-a 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.177+0000 7f67f5b83700 1 -- 172.21.15.104:0/581158286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e8009c00 con 0x7f67f006f0a0 2024-09-18T14:24:00.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f5b83700 1 -- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 msgr2=0x7f67f006f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f5b83700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 0x7f67f006f500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67f009bfa0 tx=0x7f67ec007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 1 -- 172.21.15.104:0/581158286 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec005b30 con 0x7f67f006f0a0 2024-09-18T14:24:00.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: dump: 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: _finish_auth 0 2024-09-18T14:24:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 1 -- 172.21.15.104:0/581158286 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec005cb0 con 0x7f67f006f0a0 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 5 monclient: authenticate success, global_id 14202 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 10 monclient: _renew_subs 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f0096af0 con 0x7f67f006f0a0 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 1 -- 172.21.15.104:0/581158286 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec007360 con 0x7f67f006f0a0 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e57fa700 10 monclient: dump: 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.190 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67e77fe700 4 set_mon_vals no callback set 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 10 monclient: _renew_subs 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f0135f00 con 0x7f67f006f0a0 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.181+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f00633b0 con 0x7f67f006f0a0 2024-09-18T14:24:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.185+0000 7f67e57fa700 1 -- 172.21.15.104:0/581158286 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f67ec013410 con 0x7f67f006f0a0 2024-09-18T14:24:00.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.185+0000 7f67e57fa700 1 --2- 172.21.15.104:0/581158286 >> v2:172.21.15.174:6812/17603 conn(0x7f67d40368b0 0x7f67d4038d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.185+0000 7f67e57fa700 1 -- 172.21.15.104:0/581158286 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f67ec047680 con 0x7f67f006f0a0 2024-09-18T14:24:00.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.185+0000 7f67f5382700 1 --2- 172.21.15.104:0/581158286 >> v2:172.21.15.174:6812/17603 conn(0x7f67d40368b0 0x7f67d4038d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.185+0000 7f67f5382700 10 monclient: get_auth_request con 0x7f67d40368b0 auth_method 0 2024-09-18T14:24:00.192 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.189+0000 7f67f5382700 1 --2- 172.21.15.104:0/581158286 >> v2:172.21.15.174:6812/17603 conn(0x7f67d40368b0 0x7f67d4038d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67e0007fd0 tx=0x7f67e0007b20 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.189+0000 7f67e57fa700 1 -- 172.21.15.104:0/581158286 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f67ec01a900 con 0x7f67f006f0a0 2024-09-18T14:24:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.189+0000 7f67e57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:24:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.189+0000 7f67e57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.413+0000 7f67f6384700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.413+0000 7f67f6384700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.413+0000 7f67f6384700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.413+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f67f0068a50 con 0x7f67f006f0a0 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.413+0000 7f67e57fa700 1 -- 172.21.15.104:0/581158286 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f67ec011360 con 0x7f67f006f0a0 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.413+0000 7f67e57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.413+0000 7f67e57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:24:00.418 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:24:00.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 >> v2:172.21.15.174:6812/17603 conn(0x7f67d40368b0 msgr2=0x7f67d4038d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 --2- 172.21.15.104:0/581158286 >> v2:172.21.15.174:6812/17603 conn(0x7f67d40368b0 0x7f67d4038d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67e0007fd0 tx=0x7f67e0007b20 comp rx=0 tx=0).stop 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 10 monclient: shutdown 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 msgr2=0x7f67f006f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 0x7f67f006f500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67f009bfa0 tx=0x7f67ec007c70 comp rx=0 tx=0).stop 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 shutdown_connections 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 --2- 172.21.15.104:0/581158286 >> v2:172.21.15.174:6812/17603 conn(0x7f67d40368b0 0x7f67d4038d60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67f006f0a0 0x7f67f006f500 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67f012fb10 0x7f67f006eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 --2- 172.21.15.104:0/581158286 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f67d40063d0 0x7f67f006e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 >> 172.21.15.104:0/581158286 conn(0x7f67f00a12c0 msgr2=0x7f67f009dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 shutdown_connections 2024-09-18T14:24:00.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.417+0000 7f67f6384700 1 -- 172.21.15.104:0/581158286 wait complete. 2024-09-18T14:24:00.432 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T14:24:00.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T14:24:00.432 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:24:00.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient: get_monmap_and_config 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient: build_initial_monmap 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient: monmap: 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:24:00.537907+0000 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:24:00.537907+0000 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:24:00.538 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:24:00.539 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 Processor -- start 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 -- start start 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient: init 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient: _reopen_session rank -1 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f74b809e830 0x7f74b8131d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): picked mon.noname-a con 0x7f74b809e830 addr v2:172.21.15.104:3300/0 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b8132260 0x7f74b809ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): picked mon.noname-b con 0x7f74b8132260 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:24:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f74b809f140 0x7f74b809fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): picked mon.noname-c con 0x7f74b809f140 addr v2:172.21.15.174:3300/0 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b809b7a0 con 0x7f74b809e830 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b8099f10 con 0x7f74b809f140 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b809b4c0 con 0x7f74b8132260 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): _renew_subs 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74bd745700 10 monclient(hunting): authenticate will time out at 961.527802s 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b77fe700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b8132260 msgr2=0x7f74b809ec00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b77fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b8132260 0x7f74b809ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:24:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f74b809e830 0x7f74b8131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f74b809e830 0x7f74b8131d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51308/0 (socket says 172.21.15.104:51308) 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b7fff700 1 -- 172.21.15.104:0/3293391921 learned_addr learned my addr 172.21.15.104:0/3293391921 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b7fff700 10 monclient(hunting): get_auth_request con 0x7f74b809e830 auth_method 0 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.533+0000 7f74b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 1 --2- 172.21.15.104:0/3293391921 >> v2:172.21.15.174:3300/0 conn(0x7f74b809f140 0x7f74b809fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 10 monclient(hunting): get_auth_request con 0x7f74b809f140 auth_method 0 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bcf44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 10 monclient(hunting): handle_auth_done global_id 14205 payload 274 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 -- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b8132260 msgr2=0x7f74b809ec00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 --2- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b8132260 0x7f74b809ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 -- 172.21.15.104:0/3293391921 >> v2:172.21.15.174:3300/0 conn(0x7f74b809f140 msgr2=0x7f74b809fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 --2- 172.21.15.104:0/3293391921 >> v2:172.21.15.174:3300/0 conn(0x7f74b809f140 0x7f74b809fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 10 monclient: _finish_hunting 0 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 monclient: found mon.noname-a 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 -- 172.21.15.104:0/3293391921 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b809ba80 con 0x7f74b809e830 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 -- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b809e830 msgr2=0x7f74b8131d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b7fff700 1 --2- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b809e830 0x7f74b8131d20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f74b809c460 tx=0x7f74b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dce86808c87a6fd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 1 -- 172.21.15.104:0/3293391921 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b000e070 con 0x7f74b809e830 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: dump: 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: _finish_auth 0 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 5 monclient: authenticate success, global_id 14205 2024-09-18T14:24:00.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 1 -- 172.21.15.104:0/3293391921 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b000bc30 con 0x7f74b809e830 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 10 monclient: get_monmap_and_config success 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 4 set_mon_vals no callback set 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 1 -- 172.21.15.104:0/3293391921 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b00092e0 con 0x7f74b809e830 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74b6ffd700 10 monclient: dump: 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.548 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.549 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.549 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.549 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.549 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.549 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 10 monclient: shutdown 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 -- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b809e830 msgr2=0x7f74b8131d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 --2- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b809e830 0x7f74b8131d20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f74b809c460 tx=0x7f74b0009840 comp rx=0 tx=0).stop 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 -- 172.21.15.104:0/3293391921 shutdown_connections 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 --2- 172.21.15.104:0/3293391921 >> v2:172.21.15.174:3300/0 conn(0x7f74b809f140 0x7f74b809fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 --2- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b8132260 0x7f74b809ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 --2- 172.21.15.104:0/3293391921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b809e830 0x7f74b8131d20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 -- 172.21.15.104:0/3293391921 >> 172.21.15.104:0/3293391921 conn(0x7f74b80676a0 msgr2=0x7f74b8067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 -- 172.21.15.104:0/3293391921 shutdown_connections 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 1 -- 172.21.15.104:0/3293391921 wait complete. 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 10 monclient: build_initial_monmap 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.537+0000 7f74bd745700 10 monclient: monmap: 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:24:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 Processor -- start 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 -- start start 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient: init 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient: _reopen_session rank -1 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:24:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74b809e830 0x7f74b8130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): picked mon.noname-b con 0x7f74b809e830 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b809f140 0x7f74b8141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): picked mon.noname-c con 0x7f74b809f140 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8132260 0x7f74b81415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): picked mon.noname-a con 0x7f74b8132260 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b811cd90 con 0x7f74b8132260 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b811f3e0 con 0x7f74b809e830 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b809f5e0 con 0x7f74b809f140 2024-09-18T14:24:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): _renew_subs 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient(hunting): authenticate will time out at 961.532762s 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b77fe700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b809f140 msgr2=0x7f74b8141090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b77fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b809f140 0x7f74b8141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8132260 0x7f74b81415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8132260 0x7f74b81415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51320/0 (socket says 172.21.15.104:51320) 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 -- 172.21.15.104:0/2202612129 learned_addr learned my addr 172.21.15.104:0/2202612129 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): get_auth_request con 0x7f74b8132260 auth_method 0 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74b809e830 0x7f74b8130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 10 monclient(hunting): get_auth_request con 0x7f74b809e830 auth_method 0 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient(hunting): handle_auth_done global_id 14208 payload 824 2024-09-18T14:24:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b809f140 msgr2=0x7f74b8141090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f74b809f140 0x7f74b8141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74b809e830 msgr2=0x7f74b8130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74b809e830 0x7f74b8130cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient: _finish_hunting 0 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 monclient: found mon.noname-a 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 -- 172.21.15.104:0/2202612129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0009ae0 con 0x7f74b8132260 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8132260 msgr2=0x7f74b81415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bcf44700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8132260 0x7f74b81415d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00ea80 tx=0x7f74ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0096b0 con 0x7f74b8132260 2024-09-18T14:24:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: dump: 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: _finish_auth 0 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 5 monclient: authenticate success, global_id 14208 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient: _renew_subs 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac00ca60 con 0x7f74b8132260 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b811f100 con 0x7f74b8132260 2024-09-18T14:24:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:24:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac00ce20 con 0x7f74b8132260 2024-09-18T14:24:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f749ffff700 10 monclient: dump: 2024-09-18T14:24:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74b5ffb700 4 set_mon_vals no callback set 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient: _renew_subs 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.541+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b811df30 con 0x7f74b8132260 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74bd745700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:24:00.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74bd745700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 0x7f74b811b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8064840 con 0x7f74b8131230 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f74ac013680 con 0x7f74b8132260 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 0x7f74b811b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient: get_auth_request con 0x7f74b8131230 auth_method 0 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 10 monclient(hunting): handle_auth_done global_id 14211 payload 824 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 1 --2- 172.21.15.104:0/2202612129 >> v2:172.21.15.174:6812/17603 conn(0x7f74a4035a30 0x7f74a4037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 1 -- 172.21.15.104:0/2202612129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74b8063450 con 0x7f74b8131230 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f74ac0482a0 con 0x7f74b8132260 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b77fe700 1 --2- 172.21.15.104:0/2202612129 >> v2:172.21.15.174:6812/17603 conn(0x7f74a4035a30 0x7f74a4037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b77fe700 10 monclient: get_auth_request con 0x7f74a4035a30 auth_method 0 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 msgr2=0x7f74b811b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f74b7fff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 0x7f74b811b920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74b0015000 tx=0x7f74b0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0007b50 con 0x7f74b8131230 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 10 monclient: dump: 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f74b0007cb0 con 0x7f74b8131230 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.545+0000 7f749ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.549+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 msgr2=0x7f74b811b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.549+0000 7f749ffff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 0x7f74b811b920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74b0015000 tx=0x7f74b0018000 comp rx=0 tx=0).stop 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.549+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 msgr2=0x7f74b811b920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.549+0000 7f749ffff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8131230 0x7f74b811b920 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74b0015000 tx=0x7f74b0018000 comp rx=0 tx=0).stop 2024-09-18T14:24:00.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.549+0000 7f74b77fe700 1 --2- 172.21.15.104:0/2202612129 >> v2:172.21.15.174:6812/17603 conn(0x7f74a4035a30 0x7f74a4037ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006fd0 tx=0x7f74a8006e10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74bd745700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:24:00.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74bd745700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 0x7f74b811c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74bd745700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b811f570 con 0x7f74b811be60 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 0x7f74b811c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient: get_auth_request con 0x7f74b811be60 auth_method 0 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 10 monclient(hunting): handle_auth_done global_id 14214 payload 824 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 1 -- 172.21.15.104:0/2202612129 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f74b811c7a0 con 0x7f74b811be60 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 msgr2=0x7f74b811c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f74b7fff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 0x7f74b811c260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009880 tx=0x7f74b0020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0027990 con 0x7f74b811be60 2024-09-18T14:24:00.572 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f749ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f749ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.557+0000 7f749ffff700 10 monclient: dump: 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f74b0027af0 con 0x7f74b811be60 2024-09-18T14:24:00.573 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f749ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f749ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 msgr2=0x7f74b811c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f749ffff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 0x7f74b811c260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009880 tx=0x7f74b0020040 comp rx=0 tx=0).stop 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f749ffff700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 msgr2=0x7f74b811c260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f749ffff700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b811be60 0x7f74b811c260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 24, 2024-09-18T14:24:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: 1 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 1, 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:24:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:24:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:24:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:24:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:24:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:24:00.581 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:24:00.582 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:24:00.583 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.586 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:24:00.586 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 >> v2:172.21.15.174:6812/17603 conn(0x7f74a4035a30 msgr2=0x7f74a4037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 --2- 172.21.15.104:0/2202612129 >> v2:172.21.15.174:6812/17603 conn(0x7f74a4035a30 0x7f74a4037ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006fd0 tx=0x7f74a8006e10 comp rx=0 tx=0).stop 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 10 monclient: shutdown 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8132260 msgr2=0x7f74b81415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 --2- 172.21.15.104:0/2202612129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b8132260 0x7f74b81415d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00ea80 tx=0x7f74ac0091a0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bcf44700 1 -- 172.21.15.104:0/2202612129 reap_dead start 2024-09-18T14:24:00.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bcf44700 1 -- 172.21.15.104:0/2202612129 reap_dead start 2024-09-18T14:24:00.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 shutdown_connections 2024-09-18T14:24:00.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 >> 172.21.15.104:0/2202612129 conn(0x7f74b80676a0 msgr2=0x7f74b809cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:24:00.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 shutdown_connections 2024-09-18T14:24:00.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.561+0000 7f74bd745700 1 -- 172.21.15.104:0/2202612129 wait complete. 2024-09-18T14:24:00.588 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T14:24:00.588 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T14:24:00.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:24:00.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient: get_monmap_and_config 2024-09-18T14:24:00.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient: build_initial_monmap 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient: monmap: 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:24:00.697926+0000 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:24:00.697926+0000 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:24:00.698 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 1 Processor -- start 2024-09-18T14:24:00.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 1 -- start start 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient: init 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient: _reopen_session rank -1 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f81f8097c60 0x7f81f812fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient(hunting): picked mon.noname-a con 0x7f81f8097c60 addr v2:172.21.15.104:3300/0 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81f809adf0 addr v1:172.21.15.104:6789/0 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809b1c0 0x7f81f809c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient(hunting): picked mon.noname-b con 0x7f81f809b1c0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f809db70 con 0x7f81f809adf0 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f8099300 con 0x7f81f8097c60 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f8098030 con 0x7f81f809b1c0 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient(hunting): _renew_subs 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81ff236700 10 monclient(hunting): authenticate will time out at 961.687890s 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81fd7d3700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809b1c0 msgr2=0x7f81f809c390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.693+0000 7f81fd7d3700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809b1c0 0x7f81f809c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f81f8097c60 0x7f81f812fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81effff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f81f809adf0 0x7f81f809bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57498/0 (socket says 172.21.15.104:57498) 2024-09-18T14:24:00.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81effff700 1 -- 172.21.15.104:0/2405239987 learned_addr learned my addr 172.21.15.104:0/2405239987 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): get_auth_request con 0x7f81f8097c60 auth_method 0 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_done global_id 14217 payload 274 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 -- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809b1c0 msgr2=0x7f81f809c390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 --2- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809b1c0 0x7f81f809c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 -- 172.21.15.104:0/2405239987 >> v1:172.21.15.104:6789/0 conn(0x7f81f809adf0 legacy=0x7f81f809bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient: _finish_hunting 0 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 monclient: found mon.noname-a 2024-09-18T14:24:00.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:24:00.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 -- 172.21.15.104:0/2405239987 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f809cfa0 con 0x7f81f8097c60 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 -- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 msgr2=0x7f81f812fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81fcfd2700 1 --2- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f812fb50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098e0 tx=0x7f81f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c0aad2f51bb402 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 1 -- 172.21.15.104:0/2405239987 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f400ba70 con 0x7f81f8097c60 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: dump: 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.709 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: _finish_auth 0 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 1 -- 172.21.15.104:0/2405239987 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f400bbf0 con 0x7f81f8097c60 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ff236700 5 monclient: authenticate success, global_id 14217 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ff236700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ff236700 10 monclient: get_monmap_and_config success 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 1 -- 172.21.15.104:0/2405239987 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f4011420 con 0x7f81f8097c60 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ff236700 4 set_mon_vals no callback set 2024-09-18T14:24:00.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.697+0000 7f81ef7fe700 10 monclient: dump: 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient: shutdown 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:24:00.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 msgr2=0x7f81f812fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 --2- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f812fb50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098e0 tx=0x7f81f40091a0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- 172.21.15.104:0/2405239987 shutdown_connections 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 --2- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809b1c0 0x7f81f809c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 --2- 172.21.15.104:0/2405239987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f812fb50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- 172.21.15.104:0/2405239987 >> 172.21.15.104:0/2405239987 conn(0x7f81f80a1290 msgr2=0x7f81f80a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- 172.21.15.104:0/2405239987 shutdown_connections 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- 172.21.15.104:0/2405239987 wait complete. 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient: build_initial_monmap 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient: monmap: 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:24:00.712 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 Processor -- start 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- start start 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient: init 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient: _reopen_session rank -1 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f8095a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): picked mon.noname-a con 0x7f81f8097c60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:24:00.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809adf0 0x7f81f8095fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): picked mon.noname-c con 0x7f81f809adf0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f809b1c0 0x7f81f8096500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): picked mon.noname-b con 0x7f81f809b1c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f811e850 con 0x7f81f8097c60 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f806e480 con 0x7f81f809b1c0 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81effff700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809adf0 msgr2=0x7f81f8095fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f811e570 con 0x7f81f809adf0 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): _renew_subs 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81ff236700 10 monclient(hunting): authenticate will time out at 961.695014s 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81effff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809adf0 0x7f81f8095fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:24:00.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f8095a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f8095a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:51346/0 (socket says 172.21.15.104:51346) 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 -- 172.21.15.104:0/668063360 learned_addr learned my addr 172.21.15.104:0/668063360 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): get_auth_request con 0x7f81f8097c60 auth_method 0 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fd7d3700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f809b1c0 0x7f81f8096500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fd7d3700 10 monclient(hunting): get_auth_request con 0x7f81f809b1c0 auth_method 0 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fd7d3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fd7d3700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fd7d3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient(hunting): handle_auth_done global_id 14220 payload 824 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809adf0 msgr2=0x7f81f8095fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f81f809adf0 0x7f81f8095fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f809b1c0 msgr2=0x7f81f8096500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f809b1c0 0x7f81f8096500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient: _finish_hunting 0 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 monclient: found mon.noname-a 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.701+0000 7f81fcfd2700 1 -- 172.21.15.104:0/668063360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4009420 con 0x7f81f8097c60 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81fcfd2700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 msgr2=0x7f81f8095a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81fcfd2700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f8095a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81f400b3c0 tx=0x7f81f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f401b680 con 0x7f81f8097c60 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: dump: 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: _finish_auth 0 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 5 monclient: authenticate success, global_id 14220 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f401b7e0 con 0x7f81f8097c60 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f401bba0 con 0x7f81f8097c60 2024-09-18T14:24:00.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 10 monclient: _renew_subs 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f80976e0 con 0x7f81f8097c60 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ee7fc700 4 set_mon_vals no callback set 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ecff9700 10 monclient: dump: 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.722 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 10 monclient: _renew_subs 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f8096a40 con 0x7f81f8097c60 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 0x7f81f805e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.705+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f80686b0 con 0x7f81f802ae80 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 0x7f81f805e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient: get_auth_request con 0x7f81f802ae80 auth_method 0 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f81f40193b0 con 0x7f81f8097c60 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 10 monclient(hunting): handle_auth_done global_id 4900 payload 824 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 1 -- 172.21.15.104:0/668063360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81f811d820 con 0x7f81f802ae80 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81ecff9700 1 --2- 172.21.15.104:0/668063360 >> v2:172.21.15.174:6812/17603 conn(0x7f81e4035a30 0x7f81e4037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f81f40499e0 con 0x7f81f8097c60 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81fd7d3700 1 --2- 172.21.15.104:0/668063360 >> v2:172.21.15.174:6812/17603 conn(0x7f81e4035a30 0x7f81e4037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81fd7d3700 10 monclient: get_auth_request con 0x7f81e4035a30 auth_method 0 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 msgr2=0x7f81f805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81effff700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 0x7f81f805e890 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f81f8097c10 tx=0x7f81e8007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8005e50 con 0x7f81f802ae80 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81ecff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81ecff9700 10 monclient: dump: 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.709+0000 7f81fd7d3700 1 --2- 172.21.15.104:0/668063360 >> v2:172.21.15.174:6812/17603 conn(0x7f81e4035a30 0x7f81e4037ee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81f811da70 tx=0x7f81f0009540 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.713+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f81e80070d0 con 0x7f81f802ae80 2024-09-18T14:24:00.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.713+0000 7f81ecff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:24:00.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.713+0000 7f81ecff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:24:00.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.713+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 msgr2=0x7f81f805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.713+0000 7f81ecff9700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 0x7f81f805e890 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f81f8097c10 tx=0x7f81e8007f00 comp rx=0 tx=0).stop 2024-09-18T14:24:00.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.713+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 msgr2=0x7f81f805e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:24:00.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.713+0000 7f81ecff9700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f802ae80 0x7f81f805e890 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:24:00.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81ff236700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:24:00.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81ff236700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 0x7f81f814bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:24:00.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81ff236700 10 monclient(hunting): start opening mon connection 2024-09-18T14:24:00.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f811ecc0 con 0x7f81f805edd0 2024-09-18T14:24:00.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 0x7f81f814bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient: get_auth_request con 0x7f81f805edd0 auth_method 0 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 10 monclient(hunting): handle_auth_done global_id 4903 payload 824 2024-09-18T14:24:00.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.725+0000 7f81effff700 1 -- 172.21.15.104:0/668063360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81f806c760 con 0x7f81f805edd0 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81effff700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 msgr2=0x7f81f814bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81effff700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 0x7f81f814bd30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f81f8097c10 tx=0x7f81e8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8020900 con 0x7f81f805edd0 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 10 monclient: dump: 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:24:00.732 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.733 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:24:00.733 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:24:00.733 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:24:00.733 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:24:00.733 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:24:00.733 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:24:00.733 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f81e8005c40 con 0x7f81f805edd0 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 msgr2=0x7f81f814bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 0x7f81f814bd30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f81f8097c10 tx=0x7f81e8015000 comp rx=0 tx=0).stop 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 msgr2=0x7f81f814bd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ecff9700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f805edd0 0x7f81f814bd30 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f81f8097c10 tx=0x7f81e8015000 comp rx=0 tx=0).stop 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:24:00.734 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 24, 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: 1 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 1, 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:24:00.735 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:24:00.736 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:24:00.737 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:24:00.738 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:24:00.739 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:24:00.740 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.742 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.743 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:24:00.744 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:24:00.745 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 7 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:24:00.746 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:24:00.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 >> v2:172.21.15.174:6812/17603 conn(0x7f81e4035a30 msgr2=0x7f81e4037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 --2- 172.21.15.104:0/668063360 >> v2:172.21.15.174:6812/17603 conn(0x7f81e4035a30 0x7f81e4037ee0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81f811da70 tx=0x7f81f0009540 comp rx=0 tx=0).stop 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 10 monclient: shutdown 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 msgr2=0x7f81f8095a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 --2- 172.21.15.104:0/668063360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f8097c60 0x7f81f8095a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81f400b3c0 tx=0x7f81f4016000 comp rx=0 tx=0).stop 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81fd7d3700 1 -- 172.21.15.104:0/668063360 reap_dead start 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81fd7d3700 1 -- 172.21.15.104:0/668063360 reap_dead start 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 shutdown_connections 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 >> 172.21.15.104:0/668063360 conn(0x7f81f80a1290 msgr2=0x7f81f812f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 shutdown_connections 2024-09-18T14:24:00.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:24:00.729+0000 7f81ff236700 1 -- 172.21.15.104:0/668063360 wait complete. 2024-09-18T14:24:00.751 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T14:24:00.751 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T14:24:01.702 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:01.702 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19767]: starting. 2024-09-18T14:24:01.702 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19767]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19731 2024-09-18T14:24:01.703 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19767]: created object 0... 2024-09-18T14:24:01.703 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19767]: created object 25... 2024-09-18T14:24:01.703 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19767]: created object 49... 2024-09-18T14:24:01.703 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19767]: finishing. 2024-09-18T14:24:01.703 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19767]: shutting down. 2024-09-18T14:24:01.762 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:01.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19769]: starting. 2024-09-18T14:24:01.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19769]: listing objects. 2024-09-18T14:24:01.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19769]: listed object 0... 2024-09-18T14:24:01.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19769]: listed object 25... 2024-09-18T14:24:01.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19769]: saw 50 objects 2024-09-18T14:24:01.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19769]: shutting down. 2024-09-18T14:24:02.548 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:02.548 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19768]: starting. 2024-09-18T14:24:02.548 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19768]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19731 2024-09-18T14:24:02.548 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19768]: shutting down. 2024-09-18T14:24:02.549 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:02.549 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:02.550 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:02.551 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:04.907 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19820]: starting. 2024-09-18T14:24:04.907 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19820]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19819 2024-09-18T14:24:04.908 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19820]: created object 0... 2024-09-18T14:24:04.908 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19820]: created object 25... 2024-09-18T14:24:04.908 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19820]: created object 49... 2024-09-18T14:24:04.908 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19820]: finishing. 2024-09-18T14:24:04.908 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19820]: shutting down. 2024-09-18T14:24:05.759 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19821]: starting. 2024-09-18T14:24:05.759 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19821]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19819 2024-09-18T14:24:05.759 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19821]: shutting down. 2024-09-18T14:24:07.927 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:07.927 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19854]: starting. 2024-09-18T14:24:07.927 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19854]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19819 2024-09-18T14:24:07.928 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19854]: created object 0... 2024-09-18T14:24:07.928 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19854]: created object 25... 2024-09-18T14:24:07.928 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19854]: created object 49... 2024-09-18T14:24:07.928 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19854]: finishing. 2024-09-18T14:24:07.928 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19854]: shutting down. 2024-09-18T14:24:07.987 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:07.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19856]: starting. 2024-09-18T14:24:07.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19856]: listing objects. 2024-09-18T14:24:07.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19856]: listed object 0... 2024-09-18T14:24:07.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19856]: listed object 25... 2024-09-18T14:24:07.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19856]: saw 50 objects 2024-09-18T14:24:07.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19856]: shutting down. 2024-09-18T14:24:08.785 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:08.785 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19855]: starting. 2024-09-18T14:24:08.785 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19855]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19819 2024-09-18T14:24:08.785 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19855]: shutting down. 2024-09-18T14:24:08.786 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:08.787 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:08.787 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:08.789 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:10.995 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19907]: starting. 2024-09-18T14:24:10.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19907]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19906 2024-09-18T14:24:10.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19907]: created object 0... 2024-09-18T14:24:10.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19907]: created object 25... 2024-09-18T14:24:10.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19907]: created object 49... 2024-09-18T14:24:10.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19907]: finishing. 2024-09-18T14:24:10.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19907]: shutting down. 2024-09-18T14:24:11.813 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19908]: starting. 2024-09-18T14:24:11.813 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19908]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19906 2024-09-18T14:24:11.813 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19908]: shutting down. 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19941]: starting. 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19941]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19906 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19941]: created object 0... 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19941]: created object 25... 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19941]: created object 49... 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19941]: finishing. 2024-09-18T14:24:13.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[19941]: shutting down. 2024-09-18T14:24:13.975 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:13.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19943]: starting. 2024-09-18T14:24:13.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19943]: listing objects. 2024-09-18T14:24:13.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19943]: listed object 0... 2024-09-18T14:24:13.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19943]: listed object 25... 2024-09-18T14:24:13.976 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19943]: saw 50 objects 2024-09-18T14:24:13.976 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[19943]: shutting down. 2024-09-18T14:24:14.741 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:14.741 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19942]: starting. 2024-09-18T14:24:14.741 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19942]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19906 2024-09-18T14:24:14.741 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[19942]: shutting down. 2024-09-18T14:24:14.743 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:14.743 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:14.743 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:14.745 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19994]: starting. 2024-09-18T14:24:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19994]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19993 2024-09-18T14:24:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19994]: created object 0... 2024-09-18T14:24:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19994]: created object 25... 2024-09-18T14:24:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19994]: created object 49... 2024-09-18T14:24:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19994]: finishing. 2024-09-18T14:24:16.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[19994]: shutting down. 2024-09-18T14:24:17.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19995]: starting. 2024-09-18T14:24:17.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19995]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19993 2024-09-18T14:24:17.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[19995]: shutting down. 2024-09-18T14:24:19.923 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:19.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20028]: starting. 2024-09-18T14:24:19.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20028]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-19993 2024-09-18T14:24:19.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20028]: created object 0... 2024-09-18T14:24:19.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20028]: created object 25... 2024-09-18T14:24:19.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20028]: created object 49... 2024-09-18T14:24:19.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20028]: finishing. 2024-09-18T14:24:19.924 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20028]: shutting down. 2024-09-18T14:24:19.984 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:19.984 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20030]: starting. 2024-09-18T14:24:19.984 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20030]: listing objects. 2024-09-18T14:24:19.984 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20030]: listed object 0... 2024-09-18T14:24:19.984 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20030]: listed object 25... 2024-09-18T14:24:19.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20030]: saw 50 objects 2024-09-18T14:24:19.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20030]: shutting down. 2024-09-18T14:24:21.720 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:21.720 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20029]: starting. 2024-09-18T14:24:21.721 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20029]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-19993 2024-09-18T14:24:21.721 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20029]: shutting down. 2024-09-18T14:24:21.722 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:21.723 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:21.723 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:21.725 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:22.914 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20081]: starting. 2024-09-18T14:24:22.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20081]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20080 2024-09-18T14:24:22.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20081]: created object 0... 2024-09-18T14:24:22.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20081]: created object 25... 2024-09-18T14:24:22.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20081]: created object 49... 2024-09-18T14:24:22.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20081]: finishing. 2024-09-18T14:24:22.915 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20081]: shutting down. 2024-09-18T14:24:23.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20082]: starting. 2024-09-18T14:24:23.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20082]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20080 2024-09-18T14:24:23.742 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20082]: shutting down. 2024-09-18T14:24:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20115]: starting. 2024-09-18T14:24:25.913 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20115]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20080 2024-09-18T14:24:25.913 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20115]: created object 0... 2024-09-18T14:24:25.913 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20115]: created object 25... 2024-09-18T14:24:25.913 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20115]: created object 49... 2024-09-18T14:24:25.913 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20115]: finishing. 2024-09-18T14:24:25.913 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20115]: shutting down. 2024-09-18T14:24:25.973 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:25.973 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20117]: starting. 2024-09-18T14:24:25.973 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20117]: listing objects. 2024-09-18T14:24:25.973 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20117]: listed object 0... 2024-09-18T14:24:25.973 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20117]: listed object 25... 2024-09-18T14:24:25.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20117]: saw 50 objects 2024-09-18T14:24:25.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20117]: shutting down. 2024-09-18T14:24:26.754 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:26.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20116]: starting. 2024-09-18T14:24:26.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20116]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20080 2024-09-18T14:24:26.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20116]: shutting down. 2024-09-18T14:24:26.755 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:26.755 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:26.755 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:26.757 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:29.162 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20168]: starting. 2024-09-18T14:24:29.163 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20168]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20167 2024-09-18T14:24:29.163 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20168]: created object 0... 2024-09-18T14:24:29.163 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20168]: created object 25... 2024-09-18T14:24:29.163 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20168]: created object 49... 2024-09-18T14:24:29.163 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20168]: finishing. 2024-09-18T14:24:29.163 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20168]: shutting down. 2024-09-18T14:24:29.789 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20169]: starting. 2024-09-18T14:24:29.789 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20169]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20167 2024-09-18T14:24:29.789 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20169]: shutting down. 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20202]: starting. 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20202]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20167 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20202]: created object 0... 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20202]: created object 25... 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20202]: created object 49... 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20202]: finishing. 2024-09-18T14:24:31.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20202]: shutting down. 2024-09-18T14:24:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20204]: starting. 2024-09-18T14:24:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20204]: listing objects. 2024-09-18T14:24:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20204]: listed object 0... 2024-09-18T14:24:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20204]: listed object 25... 2024-09-18T14:24:32.031 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20204]: saw 50 objects 2024-09-18T14:24:32.031 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20204]: shutting down. 2024-09-18T14:24:32.744 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:32.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20203]: starting. 2024-09-18T14:24:32.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20203]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20167 2024-09-18T14:24:32.744 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20203]: shutting down. 2024-09-18T14:24:32.745 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:32.745 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:32.745 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:32.747 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20255]: starting. 2024-09-18T14:24:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20255]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20254 2024-09-18T14:24:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20255]: created object 0... 2024-09-18T14:24:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20255]: created object 25... 2024-09-18T14:24:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20255]: created object 49... 2024-09-18T14:24:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20255]: finishing. 2024-09-18T14:24:34.898 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20255]: shutting down. 2024-09-18T14:24:35.752 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20256]: starting. 2024-09-18T14:24:35.752 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20256]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20254 2024-09-18T14:24:35.753 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20256]: shutting down. 2024-09-18T14:24:37.972 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:37.972 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20289]: starting. 2024-09-18T14:24:37.972 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20289]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20254 2024-09-18T14:24:37.972 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20289]: created object 0... 2024-09-18T14:24:37.972 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20289]: created object 25... 2024-09-18T14:24:37.973 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20289]: created object 49... 2024-09-18T14:24:37.973 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20289]: finishing. 2024-09-18T14:24:37.973 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20289]: shutting down. 2024-09-18T14:24:38.034 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:38.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20291]: starting. 2024-09-18T14:24:38.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20291]: listing objects. 2024-09-18T14:24:38.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20291]: listed object 0... 2024-09-18T14:24:38.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20291]: listed object 25... 2024-09-18T14:24:38.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20291]: saw 50 objects 2024-09-18T14:24:38.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20291]: shutting down. 2024-09-18T14:24:38.775 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:38.776 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20290]: starting. 2024-09-18T14:24:38.776 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20290]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20254 2024-09-18T14:24:38.776 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20290]: shutting down. 2024-09-18T14:24:38.777 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:38.777 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:38.777 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:38.779 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:40.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20342]: starting. 2024-09-18T14:24:40.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20342]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20341 2024-09-18T14:24:40.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20342]: created object 0... 2024-09-18T14:24:40.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20342]: created object 25... 2024-09-18T14:24:40.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20342]: created object 49... 2024-09-18T14:24:40.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20342]: finishing. 2024-09-18T14:24:40.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20342]: shutting down. 2024-09-18T14:24:41.821 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20343]: starting. 2024-09-18T14:24:41.821 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20343]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20341 2024-09-18T14:24:41.821 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20343]: shutting down. 2024-09-18T14:24:43.930 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:43.930 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20376]: starting. 2024-09-18T14:24:43.930 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20376]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20341 2024-09-18T14:24:43.930 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20376]: created object 0... 2024-09-18T14:24:43.930 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20376]: created object 25... 2024-09-18T14:24:43.931 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20376]: created object 49... 2024-09-18T14:24:43.931 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20376]: finishing. 2024-09-18T14:24:43.931 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20376]: shutting down. 2024-09-18T14:24:43.988 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:43.988 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20378]: starting. 2024-09-18T14:24:43.989 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20378]: listing objects. 2024-09-18T14:24:43.989 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20378]: listed object 0... 2024-09-18T14:24:43.989 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20378]: listed object 25... 2024-09-18T14:24:43.989 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20378]: saw 50 objects 2024-09-18T14:24:43.989 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20378]: shutting down. 2024-09-18T14:24:44.754 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:44.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20377]: starting. 2024-09-18T14:24:44.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20377]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20341 2024-09-18T14:24:44.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20377]: shutting down. 2024-09-18T14:24:44.755 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:44.756 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:44.756 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:44.757 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:46.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20429]: starting. 2024-09-18T14:24:46.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20429]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20428 2024-09-18T14:24:46.934 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20429]: created object 0... 2024-09-18T14:24:46.934 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20429]: created object 25... 2024-09-18T14:24:46.934 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20429]: created object 49... 2024-09-18T14:24:46.934 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20429]: finishing. 2024-09-18T14:24:46.934 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20429]: shutting down. 2024-09-18T14:24:47.775 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20430]: starting. 2024-09-18T14:24:47.775 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20430]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20428 2024-09-18T14:24:47.775 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20430]: shutting down. 2024-09-18T14:24:49.942 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:49.942 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20463]: starting. 2024-09-18T14:24:49.942 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20463]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20428 2024-09-18T14:24:49.942 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20463]: created object 0... 2024-09-18T14:24:49.942 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20463]: created object 25... 2024-09-18T14:24:49.942 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20463]: created object 49... 2024-09-18T14:24:49.942 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20463]: finishing. 2024-09-18T14:24:49.943 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20463]: shutting down. 2024-09-18T14:24:50.002 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20465]: starting. 2024-09-18T14:24:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20465]: listing objects. 2024-09-18T14:24:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20465]: listed object 0... 2024-09-18T14:24:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20465]: listed object 25... 2024-09-18T14:24:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20465]: saw 50 objects 2024-09-18T14:24:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20465]: shutting down. 2024-09-18T14:24:50.805 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:50.805 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20464]: starting. 2024-09-18T14:24:50.805 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20464]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20428 2024-09-18T14:24:50.805 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20464]: shutting down. 2024-09-18T14:24:50.806 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:50.806 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:50.806 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:50.808 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:24:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20516]: starting. 2024-09-18T14:24:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20516]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20515 2024-09-18T14:24:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20516]: created object 0... 2024-09-18T14:24:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20516]: created object 25... 2024-09-18T14:24:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20516]: created object 49... 2024-09-18T14:24:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20516]: finishing. 2024-09-18T14:24:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20516]: shutting down. 2024-09-18T14:24:53.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20517]: starting. 2024-09-18T14:24:53.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20517]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20515 2024-09-18T14:24:53.754 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20517]: shutting down. 2024-09-18T14:24:56.747 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:56.747 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20550]: starting. 2024-09-18T14:24:56.747 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20550]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20515 2024-09-18T14:24:56.747 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20550]: created object 0... 2024-09-18T14:24:56.748 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20550]: created object 25... 2024-09-18T14:24:56.748 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20550]: created object 49... 2024-09-18T14:24:56.748 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20550]: finishing. 2024-09-18T14:24:56.748 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20550]: shutting down. 2024-09-18T14:24:56.807 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:56.807 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20552]: starting. 2024-09-18T14:24:56.807 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20552]: listing objects. 2024-09-18T14:24:56.808 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20552]: listed object 0... 2024-09-18T14:24:56.808 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20552]: listed object 25... 2024-09-18T14:24:56.808 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20552]: saw 50 objects 2024-09-18T14:24:56.808 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20552]: shutting down. 2024-09-18T14:24:57.402 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:57.403 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20551]: starting. 2024-09-18T14:24:57.403 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20551]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20515 2024-09-18T14:24:57.403 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20551]: shutting down. 2024-09-18T14:24:57.403 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:57.404 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:24:57.404 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:24:57.405 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:00.141 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20603]: starting. 2024-09-18T14:25:00.141 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20603]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20602 2024-09-18T14:25:00.141 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20603]: created object 0... 2024-09-18T14:25:00.141 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20603]: created object 25... 2024-09-18T14:25:00.141 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20603]: created object 49... 2024-09-18T14:25:00.141 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20603]: finishing. 2024-09-18T14:25:00.141 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20603]: shutting down. 2024-09-18T14:25:00.771 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20604]: starting. 2024-09-18T14:25:00.771 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20604]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20602 2024-09-18T14:25:00.772 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20604]: shutting down. 2024-09-18T14:25:02.986 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:02.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20637]: starting. 2024-09-18T14:25:02.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20637]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20602 2024-09-18T14:25:02.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20637]: created object 0... 2024-09-18T14:25:02.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20637]: created object 25... 2024-09-18T14:25:02.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20637]: created object 49... 2024-09-18T14:25:02.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20637]: finishing. 2024-09-18T14:25:02.987 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20637]: shutting down. 2024-09-18T14:25:03.250 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:03.250 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20639]: starting. 2024-09-18T14:25:03.250 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20639]: listing objects. 2024-09-18T14:25:03.250 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20639]: listed object 0... 2024-09-18T14:25:03.250 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20639]: listed object 25... 2024-09-18T14:25:03.250 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20639]: saw 50 objects 2024-09-18T14:25:03.250 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20639]: shutting down. 2024-09-18T14:25:03.794 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:03.794 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20638]: starting. 2024-09-18T14:25:03.794 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20638]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20602 2024-09-18T14:25:03.794 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20638]: shutting down. 2024-09-18T14:25:03.795 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:03.795 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:03.795 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:03.797 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:06.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20693]: starting. 2024-09-18T14:25:06.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20693]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20692 2024-09-18T14:25:06.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20693]: created object 0... 2024-09-18T14:25:06.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20693]: created object 25... 2024-09-18T14:25:06.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20693]: created object 49... 2024-09-18T14:25:06.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20693]: finishing. 2024-09-18T14:25:06.001 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20693]: shutting down. 2024-09-18T14:25:06.829 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20694]: starting. 2024-09-18T14:25:06.830 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20694]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20692 2024-09-18T14:25:06.830 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20694]: shutting down. 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20727]: starting. 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20727]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20692 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20727]: created object 0... 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20727]: created object 25... 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20727]: created object 49... 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20727]: finishing. 2024-09-18T14:25:08.918 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20727]: shutting down. 2024-09-18T14:25:08.979 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:08.979 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20729]: starting. 2024-09-18T14:25:08.979 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20729]: listing objects. 2024-09-18T14:25:08.979 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20729]: listed object 0... 2024-09-18T14:25:08.979 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20729]: listed object 25... 2024-09-18T14:25:08.979 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20729]: saw 50 objects 2024-09-18T14:25:08.979 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20729]: shutting down. 2024-09-18T14:25:09.765 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:09.765 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20728]: starting. 2024-09-18T14:25:09.765 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20728]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20692 2024-09-18T14:25:09.765 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20728]: shutting down. 2024-09-18T14:25:09.766 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:09.766 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:09.766 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:09.769 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:11.980 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20780]: starting. 2024-09-18T14:25:11.980 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20780]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20779 2024-09-18T14:25:11.980 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20780]: created object 0... 2024-09-18T14:25:11.980 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20780]: created object 25... 2024-09-18T14:25:11.980 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20780]: created object 49... 2024-09-18T14:25:11.980 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20780]: finishing. 2024-09-18T14:25:11.981 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20780]: shutting down. 2024-09-18T14:25:12.789 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20781]: starting. 2024-09-18T14:25:12.789 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20781]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20779 2024-09-18T14:25:12.789 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20781]: shutting down. 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20814]: starting. 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20814]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20779 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20814]: created object 0... 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20814]: created object 25... 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20814]: created object 49... 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20814]: finishing. 2024-09-18T14:25:14.966 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20814]: shutting down. 2024-09-18T14:25:15.027 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:15.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20816]: starting. 2024-09-18T14:25:15.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20816]: listing objects. 2024-09-18T14:25:15.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20816]: listed object 0... 2024-09-18T14:25:15.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20816]: listed object 25... 2024-09-18T14:25:15.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20816]: saw 50 objects 2024-09-18T14:25:15.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20816]: shutting down. 2024-09-18T14:25:15.817 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:15.817 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20815]: starting. 2024-09-18T14:25:15.817 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20815]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20779 2024-09-18T14:25:15.817 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20815]: shutting down. 2024-09-18T14:25:15.818 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:15.818 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:15.818 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:15.820 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:17.922 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20867]: starting. 2024-09-18T14:25:17.923 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20867]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20866 2024-09-18T14:25:17.923 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20867]: created object 0... 2024-09-18T14:25:17.923 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20867]: created object 25... 2024-09-18T14:25:17.923 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20867]: created object 49... 2024-09-18T14:25:17.923 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20867]: finishing. 2024-09-18T14:25:17.923 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20867]: shutting down. 2024-09-18T14:25:18.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20868]: starting. 2024-09-18T14:25:18.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20868]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20866 2024-09-18T14:25:18.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20868]: shutting down. 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20901]: starting. 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20901]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20866 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20901]: created object 0... 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20901]: created object 25... 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20901]: created object 49... 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20901]: finishing. 2024-09-18T14:25:21.361 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20901]: shutting down. 2024-09-18T14:25:21.420 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:21.420 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20903]: starting. 2024-09-18T14:25:21.420 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20903]: listing objects. 2024-09-18T14:25:21.420 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20903]: listed object 0... 2024-09-18T14:25:21.421 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20903]: listed object 25... 2024-09-18T14:25:21.421 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20903]: saw 50 objects 2024-09-18T14:25:21.421 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20903]: shutting down. 2024-09-18T14:25:21.783 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:21.783 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20902]: starting. 2024-09-18T14:25:21.783 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20902]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20866 2024-09-18T14:25:21.783 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20902]: shutting down. 2024-09-18T14:25:21.784 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:21.784 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:21.784 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:21.786 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:23.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20954]: starting. 2024-09-18T14:25:23.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20954]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20953 2024-09-18T14:25:23.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20954]: created object 0... 2024-09-18T14:25:23.968 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20954]: created object 25... 2024-09-18T14:25:23.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20954]: created object 49... 2024-09-18T14:25:23.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20954]: finishing. 2024-09-18T14:25:23.969 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[20954]: shutting down. 2024-09-18T14:25:24.808 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20955]: starting. 2024-09-18T14:25:24.808 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20955]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20953 2024-09-18T14:25:24.808 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[20955]: shutting down. 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20988]: starting. 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20988]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-20953 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20988]: created object 0... 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20988]: created object 25... 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20988]: created object 49... 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20988]: finishing. 2024-09-18T14:25:27.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[20988]: shutting down. 2024-09-18T14:25:27.111 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:27.111 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20990]: starting. 2024-09-18T14:25:27.111 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20990]: listing objects. 2024-09-18T14:25:27.111 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20990]: listed object 0... 2024-09-18T14:25:27.111 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20990]: listed object 25... 2024-09-18T14:25:27.111 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20990]: saw 50 objects 2024-09-18T14:25:27.112 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[20990]: shutting down. 2024-09-18T14:25:27.762 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:27.762 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20989]: starting. 2024-09-18T14:25:27.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20989]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-20953 2024-09-18T14:25:27.763 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[20989]: shutting down. 2024-09-18T14:25:27.764 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:27.764 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:27.764 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:27.765 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:29.951 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21041]: starting. 2024-09-18T14:25:29.951 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21041]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21040 2024-09-18T14:25:29.951 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21041]: created object 0... 2024-09-18T14:25:29.951 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21041]: created object 25... 2024-09-18T14:25:29.951 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21041]: created object 49... 2024-09-18T14:25:29.951 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21041]: finishing. 2024-09-18T14:25:29.952 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21041]: shutting down. 2024-09-18T14:25:30.752 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T14:25:30.752 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T14:25:30.753 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T14:25:30.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T14:25:30.756 INFO:tasks.ceph.mon.c:Started 2024-09-18T14:25:30.756 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T14:25:30.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:30.853 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient: get_monmap_and_config 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient: build_initial_monmap 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient: monmap: 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:30.852711+0000 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:30.852711+0000 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:30.855 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:30.855 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:30.855 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:30.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 1 Processor -- start 2024-09-18T14:25:30.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 1 -- start start 2024-09-18T14:25:30.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient: init 2024-09-18T14:25:30.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0d0097c60 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 0x7fe0d009b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient(hunting): picked mon.noname-b con 0x7fe0d009adc0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe0d0134e10 0x7fe0d01371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient(hunting): picked mon.noname-c con 0x7fe0d0134e10 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d00992d0 con 0x7fe0d0097c60 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d01301f0 con 0x7fe0d0134e10 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0098030 con 0x7fe0d009adc0 2024-09-18T14:25:30.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0d4d35700 10 monclient(hunting): authenticate will time out at 1051.843306s 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 0x7fe0d009b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 0x7fe0d009b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:34752/0 (socket says 172.21.15.104:34752) 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 1 -- 172.21.15.104:0/3897978151 learned_addr learned my addr 172.21.15.104:0/3897978151 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 10 monclient(hunting): get_auth_request con 0x7fe0d009adc0 auth_method 0 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 1 --2- 172.21.15.104:0/3897978151 >> v2:172.21.15.174:3300/0 conn(0x7fe0d0134e10 0x7fe0d01371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): get_auth_request con 0x7fe0d0134e10 auth_method 0 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:30.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 1 -- 172.21.15.104:0/3897978151 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 msgr2=0x7fe0d009b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 1 -- 172.21.15.104:0/3897978151 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 msgr2=0x7fe0d009b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 1 --2- 172.21.15.104:0/3897978151 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 0x7fe0d009b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0cdd9b700 1 --2- 172.21.15.104:0/3897978151 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 0x7fe0d009b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 10 monclient(hunting): handle_auth_done global_id 5248 payload 274 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.848+0000 7fe0ced9d700 1 -- 172.21.15.104:0/3897978151 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 msgr2=0x7fe0d009b1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 1 --2- 172.21.15.104:0/3897978151 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 0x7fe0d009b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 1 -- 172.21.15.104:0/3897978151 >> v1:172.21.15.174:6789/0 conn(0x7fe0d0097c60 legacy=0x7fe0d012fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 10 monclient: _finish_hunting 0 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 1 monclient: found mon.noname-c 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:30.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:25:30.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 1 -- 172.21.15.104:0/3897978151 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d009b700 con 0x7fe0d0134e10 2024-09-18T14:25:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 1 -- 172.21.15.104:0/3897978151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d0134e10 msgr2=0x7fe0d01371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:30.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0ced9d700 1 --2- 172.21.15.104:0/3897978151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d0134e10 0x7fe0d01371f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0130390 tx=0x7fe0c8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24de1acf4fec55f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 1 -- 172.21.15.104:0/3897978151 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c800e070 con 0x7fe0d0134e10 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: dump: 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:30.866 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: _finish_auth 0 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 5 monclient: authenticate success, global_id 5248 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 1 -- 172.21.15.104:0/3897978151 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c800be40 con 0x7fe0d0134e10 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 1 -- 172.21.15.104:0/3897978151 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8012660 con 0x7fe0d0134e10 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0cd59a700 10 monclient: dump: 2024-09-18T14:25:30.867 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 4 set_mon_vals no callback set 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 10 monclient: shutdown 2024-09-18T14:25:30.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3897978151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d0134e10 msgr2=0x7fe0d01371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3897978151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d0134e10 0x7fe0d01371f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0130390 tx=0x7fe0c8009780 comp rx=0 tx=0).stop 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3897978151 shutdown_connections 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3897978151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d0134e10 0x7fe0d01371f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.852+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3897978151 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d009adc0 0x7fe0d009b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3897978151 >> 172.21.15.104:0/3897978151 conn(0x7fe0d0067d70 msgr2=0x7fe0d00a1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3897978151 shutdown_connections 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3897978151 wait complete. 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient: build_initial_monmap 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient: monmap: 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:30.869 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 Processor -- start 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 -- start start 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient: init 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 0x7fe0d012fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): picked mon.noname-a con 0x7fe0d0097c60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d009adc0 0x7fe0d0132670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:30.870 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): picked mon.noname-b con 0x7fe0d009adc0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 0x7fe0d0132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): picked mon.noname-c con 0x7fe0d0134e10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0118a60 con 0x7fe0d0097c60 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d01179a0 con 0x7fe0d009adc0 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d009f610 con 0x7fe0d0134e10 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0d4d35700 10 monclient(hunting): authenticate will time out at 1051.851542s 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 0x7fe0d0132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 0x7fe0d012fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 0x7fe0d012fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39602/0 (socket says 172.21.15.104:39602) 2024-09-18T14:25:30.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 1 -- 172.21.15.104:0/3593027031 learned_addr learned my addr 172.21.15.104:0/3593027031 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 10 monclient(hunting): get_auth_request con 0x7fe0d0097c60 auth_method 0 2024-09-18T14:25:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 10 monclient(hunting): get_auth_request con 0x7fe0d0134e10 auth_method 0 2024-09-18T14:25:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:30.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 1 -- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 msgr2=0x7fe0d0132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 1 -- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 msgr2=0x7fe0d0132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 0x7fe0d0132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ce59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:30.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0ced9d700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 0x7fe0d0132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.856+0000 7fe0cdd9b700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d009adc0 0x7fe0d0132670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0cdd9b700 10 monclient(hunting): get_auth_request con 0x7fe0d009adc0 auth_method 0 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0cdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0cdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0cdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0cdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0cdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0cdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 10 monclient(hunting): handle_auth_done global_id 14556 payload 824 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 -- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 msgr2=0x7fe0d0132bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 0x7fe0d0132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 -- 172.21.15.104:0/3593027031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d009adc0 msgr2=0x7fe0d0132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d009adc0 0x7fe0d0132670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:30.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 10 monclient: _finish_hunting 0 2024-09-18T14:25:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 monclient: found mon.noname-a 2024-09-18T14:25:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:30.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 -- 172.21.15.104:0/3593027031 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c8009a20 con 0x7fe0d0097c60 2024-09-18T14:25:30.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 -- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 msgr2=0x7fe0d012fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:30.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ce59c700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 0x7fe0d012fb80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400c600 tx=0x7fe0c4007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:30.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 1 -- 172.21.15.104:0/3593027031 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40117a0 con 0x7fe0d0097c60 2024-09-18T14:25:30.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: dump: 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: _finish_auth 0 2024-09-18T14:25:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0d4d35700 5 monclient: authenticate success, global_id 14556 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 1 -- 172.21.15.104:0/3593027031 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c4011900 con 0x7fe0d0097c60 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0d4d35700 10 monclient: _renew_subs 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0d4d35700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d0098900 con 0x7fe0d0097c60 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 1 -- 172.21.15.104:0/3593027031 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4011cc0 con 0x7fe0d0097c60 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0ccd99700 4 set_mon_vals no callback set 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0be7fc700 10 monclient: dump: 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.879 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0d4d35700 10 monclient: _renew_subs 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0d4d35700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.860+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d0117b70 con 0x7fe0d0097c60 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0be7fc700 1 -- 172.21.15.104:0/3593027031 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fe0c4012630 con 0x7fe0d0097c60 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0be7fc700 1 --2- 172.21.15.104:0/3593027031 >> v2:172.21.15.174:6812/17603 conn(0x7fe0b0035a80 0x7fe0b0037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0be7fc700 1 -- 172.21.15.104:0/3593027031 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fe0c4047de0 con 0x7fe0d0097c60 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0d4d35700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:30.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0d4d35700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0d4d35700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d0119410 con 0x7fe0d0097c60 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0cdd9b700 1 --2- 172.21.15.104:0/3593027031 >> v2:172.21.15.174:6812/17603 conn(0x7fe0b0035a80 0x7fe0b0037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.864+0000 7fe0cdd9b700 10 monclient: get_auth_request con 0x7fe0b0035a80 auth_method 0 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.868+0000 7fe0cdd9b700 1 --2- 172.21.15.104:0/3593027031 >> v2:172.21.15.174:6812/17603 conn(0x7fe0b0035a80 0x7fe0b0037f30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0133e80 tx=0x7fe0c0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.872+0000 7fe0be7fc700 1 -- 172.21.15.104:0/3593027031 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe0c4011e20 con 0x7fe0d0097c60 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.872+0000 7fe0be7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:30.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:30.872+0000 7fe0be7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:31.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:31.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:31.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:31.100 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe0d0064b30 con 0x7fe0d0097c60 2024-09-18T14:25:31.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0be7fc700 1 -- 172.21.15.104:0/3593027031 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fe0c402a610 con 0x7fe0d0097c60 2024-09-18T14:25:31.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0be7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:31.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0be7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:31.101 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:31.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 >> v2:172.21.15.174:6812/17603 conn(0x7fe0b0035a80 msgr2=0x7fe0b0037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:31.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3593027031 >> v2:172.21.15.174:6812/17603 conn(0x7fe0b0035a80 0x7fe0b0037f30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0133e80 tx=0x7fe0c0008040 comp rx=0 tx=0).stop 2024-09-18T14:25:31.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 10 monclient: shutdown 2024-09-18T14:25:31.103 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 msgr2=0x7fe0d012fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 0x7fe0d012fb80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400c600 tx=0x7fe0c4007f60 comp rx=0 tx=0).stop 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 shutdown_connections 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3593027031 >> v2:172.21.15.174:6812/17603 conn(0x7fe0b0035a80 0x7fe0b0037f30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe0d0134e10 0x7fe0d0132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0d009adc0 0x7fe0d0132670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 --2- 172.21.15.104:0/3593027031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0d0097c60 0x7fe0d012fb80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 >> 172.21.15.104:0/3593027031 conn(0x7fe0d0067d70 msgr2=0x7fe0d009bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 shutdown_connections 2024-09-18T14:25:31.104 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:31.096+0000 7fe0d4d35700 1 -- 172.21.15.104:0/3593027031 wait complete. 2024-09-18T14:25:31.786 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21042]: starting. 2024-09-18T14:25:31.786 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21042]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21040 2024-09-18T14:25:31.786 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21042]: shutting down. 2024-09-18T14:25:34.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21075]: starting. 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21075]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21040 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21075]: created object 0... 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21075]: created object 25... 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21075]: created object 49... 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21075]: finishing. 2024-09-18T14:25:34.142 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21075]: shutting down. 2024-09-18T14:25:34.201 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:34.201 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21077]: starting. 2024-09-18T14:25:34.201 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21077]: listing objects. 2024-09-18T14:25:34.202 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21077]: listed object 0... 2024-09-18T14:25:34.202 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21077]: listed object 25... 2024-09-18T14:25:34.202 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21077]: saw 50 objects 2024-09-18T14:25:34.202 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21077]: shutting down. 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.220+0000 7fafa541e700 10 monclient: get_monmap_and_config 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.220+0000 7fafa541e700 10 monclient: build_initial_monmap 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.220+0000 7fafa541e700 10 monclient: monmap: 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:34.227037+0000 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:34.227037+0000 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:34.228 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 1 Processor -- start 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 1 -- start start 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient: init 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient(hunting): picked mon.noname-c con 0x7fafa009fe10 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafa009c900 addr v1:172.21.15.104:6789/0 2024-09-18T14:25:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafa009c090 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa009b990 con 0x7fafa009c900 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa009b6b0 con 0x7fafa009c090 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa0132a00 con 0x7fafa009fe10 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7fafa541e700 10 monclient(hunting): authenticate will time out at 1055.217280s 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf97fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fafa009c900 0x7fafa0131be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34648/0 (socket says 172.21.15.104:34648) 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf97fff700 1 -- 172.21.15.104:0/3842257514 learned_addr learned my addr 172.21.15.104:0/3842257514 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): get_auth_request con 0x7fafa009fe10 auth_method 0 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient(hunting): handle_auth_done global_id 5272 payload 274 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 >> v1:172.21.15.174:6789/0 conn(0x7fafa009c090 legacy=0x7fafa01322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 >> v1:172.21.15.104:6789/0 conn(0x7fafa009c900 legacy=0x7fafa0131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient: _finish_hunting 0 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 monclient: found mon.noname-c 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa009a060 con 0x7fafa009fe10 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 msgr2=0x7fafa00a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:25:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 msgr2=0x7fafa00a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:25:34.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7faf900098e0 tx=0x7faf900091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:34.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.224+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient: get_auth_request con 0x7fafa009fe10 auth_method 0 2024-09-18T14:25:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 0 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:25:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient: ms_handle_reset current mon v2:172.21.15.174:3300/0 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 1 -- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 msgr2=0x7fafa00a01e0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.174:3300/0 conn(0x7fafa009fe10 0x7fafa00a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient: _add_conns ranks=[3,2] 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.104:3300/0 conn(0x7faf8c003550 0x7faf8c0059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient(hunting): picked mon.noname-a con 0x7faf8c003550 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 1 --2- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faf8c006290 0x7faf8c008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient(hunting): picked mon.noname-b con 0x7faf8c006290 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 1 -- 172.21.15.104:0/3842257514 --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8c0090c0 con 0x7faf8c003550 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 1 -- 172.21.15.104:0/3842257514 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c009230 con 0x7faf8c006290 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9f7fe700 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.174:3300/0 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> v2:172.21.15.104:3300/0 conn(0x7faf8c003550 0x7faf8c0059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 1 --2- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faf8c006290 0x7faf8c008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): get_auth_request con 0x7faf8c003550 auth_method 0 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 10 monclient(hunting): get_auth_request con 0x7faf8c006290 auth_method 0 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient(hunting): handle_auth_done global_id 5272 payload 293 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faf8c006290 msgr2=0x7faf8c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faf8c006290 0x7faf8c008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient: _finish_hunting 0 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 1 monclient: found mon.noname-a 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0094b0 con 0x7faf8c003550 2024-09-18T14:25:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.428+0000 7faf97fff700 1 --2- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faf8c006290 0x7faf8c008740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:25:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9ffff700 1 -- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c003550 msgr2=0x7faf8c0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9ffff700 1 --2- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c003550 0x7faf8c0059c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faf900093f0 tx=0x7faf900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ddc81caf83bc00 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 1 -- 172.21.15.104:0/3842257514 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90009e80 con 0x7faf8c003550 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: dump: 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: _finish_auth 0 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 1 -- 172.21.15.104:0/3842257514 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf90015070 con 0x7faf8c003550 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 5 monclient: authenticate success, global_id 5272 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 4 set_mon_vals no callback set 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 1 -- 172.21.15.104:0/3842257514 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9001c440 con 0x7faf8c003550 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7faf9f7fe700 10 monclient: dump: 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 10 monclient: shutdown 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 1 -- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c003550 msgr2=0x7faf8c0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 1 --2- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c003550 0x7faf8c0059c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faf900093f0 tx=0x7faf900091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa4c1d700 1 -- 172.21.15.104:0/3842257514 reap_dead start 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 1 -- 172.21.15.104:0/3842257514 shutdown_connections 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 1 --2- 172.21.15.104:0/3842257514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c003550 0x7faf8c0059c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 1 -- 172.21.15.104:0/3842257514 >> 172.21.15.104:0/3842257514 conn(0x7fafa00a6080 msgr2=0x7fafa00676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 1 -- 172.21.15.104:0/3842257514 shutdown_connections 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 1 -- 172.21.15.104:0/3842257514 wait complete. 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 10 monclient: build_initial_monmap 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.432+0000 7fafa541e700 10 monclient: monmap: 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:34.444 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 Processor -- start 2024-09-18T14:25:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 -- start start 2024-09-18T14:25:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient: init 2024-09-18T14:25:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf8c003550 0x7fafa006edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): picked mon.noname-b con 0x7faf8c003550 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 0x7fafa0130ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): picked mon.noname-a con 0x7fafa011bf50 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 0x7fafa0141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): picked mon.noname-c con 0x7fafa0131420 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa009ae40 con 0x7fafa011bf50 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa0131d10 con 0x7faf8c003550 2024-09-18T14:25:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa011e370 con 0x7fafa0131420 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa541e700 10 monclient(hunting): authenticate will time out at 1055.430524s 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 0x7fafa0130ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 0x7fafa0141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 0x7fafa0141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:34792/0 (socket says 172.21.15.104:34792) 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf8c003550 0x7fafa006edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 0x7fafa0130ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39640/0 (socket says 172.21.15.104:39640) 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 1 -- 172.21.15.104:0/1399085435 learned_addr learned my addr 172.21.15.104:0/1399085435 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 10 monclient(hunting): get_auth_request con 0x7fafa0131420 auth_method 0 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:34.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): get_auth_request con 0x7fafa011bf50 auth_method 0 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 10 monclient(hunting): get_auth_request con 0x7faf8c003550 auth_method 0 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 1 -- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 msgr2=0x7fafa0141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 1 -- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 msgr2=0x7fafa0141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 0x7fafa0141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7fafa4c1d700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 0x7fafa0141430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient(hunting): handle_auth_done global_id 14571 payload 824 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 -- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 msgr2=0x7fafa0141430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 0x7fafa0141430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 -- 172.21.15.104:0/1399085435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf8c003550 msgr2=0x7fafa006edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf8c003550 0x7fafa006edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient: _finish_hunting 0 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 monclient: found mon.noname-a 2024-09-18T14:25:34.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf97fff700 1 -- 172.21.15.104:0/1399085435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa006f890 con 0x7fafa011bf50 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.436+0000 7faf9ffff700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf8c003550 0x7fafa006edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf97fff700 1 -- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 msgr2=0x7fafa0130ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf97fff700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 0x7fafa0130ee0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf8800fa80 tx=0x7faf88009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 1 -- 172.21.15.104:0/1399085435 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88004f10 con 0x7fafa011bf50 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: dump: 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: _finish_auth 0 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 1 -- 172.21.15.104:0/1399085435 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88005d40 con 0x7fafa011bf50 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 1 -- 172.21.15.104:0/1399085435 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88007320 con 0x7fafa011bf50 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7fafa541e700 5 monclient: authenticate success, global_id 14571 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7fafa541e700 10 monclient: _renew_subs 2024-09-18T14:25:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9effd700 4 set_mon_vals no callback set 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7fafa541e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa011cc30 con 0x7fafa011bf50 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7faf9cff9700 10 monclient: dump: 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:34.454 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:34.455 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:34.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7fafa541e700 10 monclient: _renew_subs 2024-09-18T14:25:34.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7fafa541e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:34.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.440+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa011d0a0 con 0x7fafa011bf50 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7faf9cff9700 1 -- 172.21.15.104:0/1399085435 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7faf8800d640 con 0x7fafa011bf50 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7fafa541e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7fafa541e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7fafa541e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa011e520 con 0x7fafa011bf50 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7faf9cff9700 1 --2- 172.21.15.104:0/1399085435 >> v2:172.21.15.174:6812/17603 conn(0x7faf80035a80 0x7faf80037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7faf9cff9700 1 -- 172.21.15.104:0/1399085435 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 3894+0+0 (secure 0 0 0) 0x7faf880482d0 con 0x7fafa011bf50 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7faf9ffff700 1 --2- 172.21.15.104:0/1399085435 >> v2:172.21.15.174:6812/17603 conn(0x7faf80035a80 0x7faf80037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.444+0000 7faf9ffff700 10 monclient: get_auth_request con 0x7faf80035a80 auth_method 0 2024-09-18T14:25:34.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.448+0000 7faf9ffff700 1 --2- 172.21.15.104:0/1399085435 >> v2:172.21.15.174:6812/17603 conn(0x7faf80035a80 0x7faf80037f30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf9001d000 tx=0x7faf9001b000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:34.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.452+0000 7faf9cff9700 1 -- 172.21.15.104:0/1399085435 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fafa011e520 con 0x7fafa011bf50 2024-09-18T14:25:34.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.452+0000 7faf9cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:34.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.452+0000 7faf9cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.680+0000 7fafa541e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.680+0000 7fafa541e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.680+0000 7fafa541e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.680+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fafa011e520 con 0x7fafa011bf50 2024-09-18T14:25:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7faf9cff9700 1 -- 172.21.15.104:0/1399085435 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fafa011e520 con 0x7fafa011bf50 2024-09-18T14:25:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7faf9cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7faf9cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:34.688 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":95,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 >> v2:172.21.15.174:6812/17603 conn(0x7faf80035a80 msgr2=0x7faf80037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 --2- 172.21.15.104:0/1399085435 >> v2:172.21.15.174:6812/17603 conn(0x7faf80035a80 0x7faf80037f30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf9001d000 tx=0x7faf9001b000 comp rx=0 tx=0).stop 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 10 monclient: shutdown 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 msgr2=0x7fafa0130ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 0x7fafa0130ee0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf8800fa80 tx=0x7faf88009040 comp rx=0 tx=0).stop 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 shutdown_connections 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 --2- 172.21.15.104:0/1399085435 >> v2:172.21.15.174:6812/17603 conn(0x7faf80035a80 0x7faf80037f30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fafa0131420 0x7fafa0141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa011bf50 0x7fafa0130ee0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 --2- 172.21.15.104:0/1399085435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf8c003550 0x7fafa006edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 >> 172.21.15.104:0/1399085435 conn(0x7fafa00a6080 msgr2=0x7fafa0131f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 shutdown_connections 2024-09-18T14:25:34.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:34.684+0000 7fafa541e700 1 -- 172.21.15.104:0/1399085435 wait complete. 2024-09-18T14:25:34.954 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:34.955 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21076]: starting. 2024-09-18T14:25:34.955 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21076]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21040 2024-09-18T14:25:34.955 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21076]: shutting down. 2024-09-18T14:25:34.956 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:34.956 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:34.956 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:34.957 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:37.204 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21128]: starting. 2024-09-18T14:25:37.205 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21128]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21127 2024-09-18T14:25:37.205 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21128]: created object 0... 2024-09-18T14:25:37.205 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21128]: created object 25... 2024-09-18T14:25:37.205 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21128]: created object 49... 2024-09-18T14:25:37.205 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21128]: finishing. 2024-09-18T14:25:37.205 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21128]: shutting down. 2024-09-18T14:25:37.703 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:37.765 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21129]: starting. 2024-09-18T14:25:37.765 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21129]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21127 2024-09-18T14:25:37.765 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21129]: shutting down. 2024-09-18T14:25:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.807+0000 7fa95b59e700 10 monclient: get_monmap_and_config 2024-09-18T14:25:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.807+0000 7fa95b59e700 10 monclient: build_initial_monmap 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.807+0000 7fa95b59e700 10 monclient: monmap: 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:37.814248+0000 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:37.814248+0000 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:37.815 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.807+0000 7fa95b59e700 1 Processor -- start 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.807+0000 7fa95b59e700 1 -- start start 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.807+0000 7fa95b59e700 10 monclient: init 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa95c063ee0 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa95c06e110 0x7fa95c06bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa95c06e110 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa95c134040 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0642b0 con 0x7fa95c063ee0 2024-09-18T14:25:37.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c09ddc0 con 0x7fa95c06e110 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c09d4f0 con 0x7fa95c134040 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95b59e700 10 monclient(hunting): authenticate will time out at 1058.804742s 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c136420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:34802/0 (socket says 172.21.15.104:34802) 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 1 -- 172.21.15.104:0/2052668645 learned_addr learned my addr 172.21.15.104:0/2052668645 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 10 monclient(hunting): get_auth_request con 0x7fa95c134040 auth_method 0 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 1 --2- 172.21.15.104:0/2052668645 >> v2:172.21.15.104:3300/0 conn(0x7fa95c06e110 0x7fa95c06bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 1 -- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 msgr2=0x7fa95c136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 1 -- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 msgr2=0x7fa95c136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 1 --2- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c136420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): get_auth_request con 0x7fa95c06e110 auth_method 0 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa95ad9d700 1 --2- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c136420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient(hunting): handle_auth_done global_id 14595 payload 274 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 1 -- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 msgr2=0x7fa95c136420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:37.820 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 1 --2- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c136420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 1 -- 172.21.15.104:0/2052668645 >> v1:172.21.15.174:6789/0 conn(0x7fa95c063ee0 legacy=0x7fa95c06b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient: _finish_hunting 0 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 1 monclient: found mon.noname-a 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.811+0000 7fa959d9b700 1 -- 172.21.15.104:0/2052668645 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c09d930 con 0x7fa95c06e110 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.815+0000 7fa959d9b700 1 -- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c06e110 msgr2=0x7fa95c06bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.815+0000 7fa959d9b700 1 --2- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c06e110 0x7fa95c06bd60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9440098e0 tx=0x7fa9440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f69119918616737 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 1 -- 172.21.15.104:0/2052668645 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944009e80 con 0x7fa95c06e110 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: dump: 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:37.827 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: _finish_auth 0 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 5 monclient: authenticate success, global_id 14595 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 1 -- 172.21.15.104:0/2052668645 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94400ba70 con 0x7fa95c06e110 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 4 set_mon_vals no callback set 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 1 -- 172.21.15.104:0/2052668645 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94400bdf0 con 0x7fa95c06e110 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:37.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95959a700 10 monclient: dump: 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:37.829 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 10 monclient: shutdown 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 -- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c06e110 msgr2=0x7fa95c06bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c06e110 0x7fa95c06bd60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9440098e0 tx=0x7fa9440091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 -- 172.21.15.104:0/2052668645 shutdown_connections 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2052668645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c06e110 0x7fa95c06bd60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 -- 172.21.15.104:0/2052668645 >> 172.21.15.104:0/2052668645 conn(0x7fa95c068120 msgr2=0x7fa95c0a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 -- 172.21.15.104:0/2052668645 shutdown_connections 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.819+0000 7fa95b59e700 1 -- 172.21.15.104:0/2052668645 wait complete. 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient: build_initial_monmap 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient: monmap: 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:37.834 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 Processor -- start 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 -- start start 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient: init 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 0x7fa95c1324a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa95c063ee0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c1329e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa95c134040 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa95c132f20 0x7fa95c133350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa95c132f20 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:37.835 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c09a9d0 con 0x7fa95c063ee0 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c133890 con 0x7fa95c132f20 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c118270 con 0x7fa95c134040 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95b59e700 10 monclient(hunting): authenticate will time out at 1058.817002s 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 0x7fa95c1324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c1329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 0x7fa95c1324a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39678/0 (socket says 172.21.15.104:39678) 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 -- 172.21.15.104:0/2906955504 learned_addr learned my addr 172.21.15.104:0/2906955504 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): get_auth_request con 0x7fa95c063ee0 auth_method 0 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 10 monclient(hunting): get_auth_request con 0x7fa95c134040 auth_method 0 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95ad9d700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa95c132f20 0x7fa95c133350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95ad9d700 10 monclient(hunting): get_auth_request con 0x7fa95c132f20 auth_method 0 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 1 -- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 msgr2=0x7fa95c1329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 1 -- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 msgr2=0x7fa95c1329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c1329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient(hunting): handle_auth_done global_id 14598 payload 824 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa959d9b700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c1329e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 -- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 msgr2=0x7fa95c1329e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c1329e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 -- 172.21.15.104:0/2906955504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa95c132f20 msgr2=0x7fa95c133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa95c132f20 0x7fa95c133350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient: _finish_hunting 0 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 monclient: found mon.noname-a 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 -- 172.21.15.104:0/2906955504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa944009420 con 0x7fa95c063ee0 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95ad9d700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa95c132f20 0x7fa95c133350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 -- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 msgr2=0x7fa95c1324a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:37.838 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.823+0000 7fa95a59c700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 0x7fa95c1324a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00c980 tx=0x7fa94c002760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 1 -- 172.21.15.104:0/2906955504 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c00a9b0 con 0x7fa95c063ee0 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: dump: 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:37.839 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: _finish_auth 0 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 5 monclient: authenticate success, global_id 14598 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 10 monclient: _renew_subs 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c117fd0 con 0x7fa95c063ee0 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 10 monclient: _renew_subs 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c09a550 con 0x7fa95c063ee0 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 1 -- 172.21.15.104:0/2906955504 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c00ab30 con 0x7fa95c063ee0 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 1 -- 172.21.15.104:0/2906955504 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c012420 con 0x7fa95c063ee0 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa958d99700 4 set_mon_vals no callback set 2024-09-18T14:25:37.840 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 10 monclient: dump: 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:37.841 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:37.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 1 -- 172.21.15.104:0/2906955504 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fa94c0127d0 con 0x7fa95c063ee0 2024-09-18T14:25:37.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:37.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:37.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:37.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9400050b0 con 0x7fa95c063ee0 2024-09-18T14:25:37.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 1 --2- 172.21.15.104:0/2906955504 >> v2:172.21.15.174:6812/17603 conn(0x7fa93c035a80 0x7fa93c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:37.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa9527fc700 1 -- 172.21.15.104:0/2906955504 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa94c0483a0 con 0x7fa95c063ee0 2024-09-18T14:25:37.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa959d9b700 1 --2- 172.21.15.104:0/2906955504 >> v2:172.21.15.174:6812/17603 conn(0x7fa93c035a80 0x7fa93c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:37.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.827+0000 7fa959d9b700 10 monclient: get_auth_request con 0x7fa93c035a80 auth_method 0 2024-09-18T14:25:37.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.831+0000 7fa959d9b700 1 --2- 172.21.15.104:0/2906955504 >> v2:172.21.15.174:6812/17603 conn(0x7fa93c035a80 0x7fa93c037f30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa95c06e4e0 tx=0x7fa944016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:37.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.835+0000 7fa9527fc700 1 -- 172.21.15.104:0/2906955504 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa94c022d70 con 0x7fa95c063ee0 2024-09-18T14:25:37.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.835+0000 7fa9527fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:37.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:37.835+0000 7fa9527fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:38.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa9400058e0 con 0x7fa95c063ee0 2024-09-18T14:25:38.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa9527fc700 1 -- 172.21.15.104:0/2906955504 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa94c016020 con 0x7fa95c063ee0 2024-09-18T14:25:38.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa9527fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:38.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa9527fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:38.066 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":98,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:38.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 >> v2:172.21.15.174:6812/17603 conn(0x7fa93c035a80 msgr2=0x7fa93c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:38.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2906955504 >> v2:172.21.15.174:6812/17603 conn(0x7fa93c035a80 0x7fa93c037f30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa95c06e4e0 tx=0x7fa944016000 comp rx=0 tx=0).stop 2024-09-18T14:25:38.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 10 monclient: shutdown 2024-09-18T14:25:38.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:38.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 msgr2=0x7fa95c1324a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:38.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.059+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 0x7fa95c1324a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00c980 tx=0x7fa94c002760 comp rx=0 tx=0).stop 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 shutdown_connections 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2906955504 >> v2:172.21.15.174:6812/17603 conn(0x7fa93c035a80 0x7fa93c037f30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa95c132f20 0x7fa95c133350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa95c134040 0x7fa95c1329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 --2- 172.21.15.104:0/2906955504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa95c063ee0 0x7fa95c1324a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 >> 172.21.15.104:0/2906955504 conn(0x7fa95c068120 msgr2=0x7fa95c09dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 shutdown_connections 2024-09-18T14:25:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:38.063+0000 7fa95b59e700 1 -- 172.21.15.104:0/2906955504 wait complete. 2024-09-18T14:25:41.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient: get_monmap_and_config 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient: build_initial_monmap 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient: monmap: 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:41.193761+0000 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:41.193761+0000 2024-09-18T14:25:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:41.195 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 1 Processor -- start 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 1 -- start start 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient: init 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f393c095a50 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f393c095e20 0x7f393c096240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient(hunting): picked mon.noname-a con 0x7f393c095e20 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 0x7f393c136180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient(hunting): picked mon.noname-b con 0x7f393c133da0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393c096bf0 con 0x7f393c095a50 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c12f510 con 0x7f393c095e20 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c097030 con 0x7f393c133da0 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941959700 10 monclient(hunting): authenticate will time out at 1062.183347s 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f393c095e20 0x7f393c096240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 0x7f393c136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f393c095e20 0x7f393c096240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39198/0 (socket says 172.21.15.104:39198) 2024-09-18T14:25:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 1 -- 172.21.15.104:0/3808036259 learned_addr learned my addr 172.21.15.104:0/3808036259 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 10 monclient(hunting): get_auth_request con 0x7f393c095e20 auth_method 0 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 10 monclient(hunting): get_auth_request con 0x7f393c133da0 auth_method 0 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 1 -- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 msgr2=0x7f393c136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 1 -- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 msgr2=0x7f393c136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:25:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 1 --2- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 0x7f393c136180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.187+0000 7f3941158700 1 --2- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 0x7f393c136180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 10 monclient(hunting): handle_auth_done global_id 14601 payload 274 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 1 -- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 msgr2=0x7f393c136180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 1 --2- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 0x7f393c136180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 1 -- 172.21.15.104:0/3808036259 >> v1:172.21.15.174:6789/0 conn(0x7f393c095a50 legacy=0x7f393c12ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 10 monclient: _finish_hunting 0 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 1 monclient: found mon.noname-a 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 1 -- 172.21.15.104:0/3808036259 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c098530 con 0x7f393c095e20 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 1 -- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c095e20 msgr2=0x7f393c096240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393bfff700 1 --2- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c095e20 0x7f393c096240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f393c12f6b0 tx=0x7f392c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4c4916a3744247 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 1 -- 172.21.15.104:0/3808036259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c010070 con 0x7f393c095e20 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: dump: 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: _finish_auth 0 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 5 monclient: authenticate success, global_id 14601 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 1 -- 172.21.15.104:0/3808036259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c00de90 con 0x7f393c095e20 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 4 set_mon_vals no callback set 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 1 -- 172.21.15.104:0/3808036259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c009610 con 0x7f393c095e20 2024-09-18T14:25:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f393b7fe700 10 monclient: dump: 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 10 monclient: shutdown 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 -- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c095e20 msgr2=0x7f393c096240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 --2- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c095e20 0x7f393c096240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f393c12f6b0 tx=0x7f392c00ba50 comp rx=0 tx=0).stop 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 -- 172.21.15.104:0/3808036259 shutdown_connections 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 --2- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c133da0 0x7f393c136180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 --2- 172.21.15.104:0/3808036259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c095e20 0x7f393c096240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 -- 172.21.15.104:0/3808036259 >> 172.21.15.104:0/3808036259 conn(0x7f393c0a12c0 msgr2=0x7f393c0a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:41.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 -- 172.21.15.104:0/3808036259 shutdown_connections 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.191+0000 7f3941959700 1 -- 172.21.15.104:0/3808036259 wait complete. 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient: build_initial_monmap 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient: monmap: 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:41.208 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 Processor -- start 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 -- start start 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient: init 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f393c095a50 0x7f393c12ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): picked mon.noname-b con 0x7f393c095a50 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 0x7f393c12b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): picked mon.noname-c con 0x7f393c095e20 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 0x7f393c12b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): picked mon.noname-a con 0x7f393c133da0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c09b370 con 0x7f393c133da0 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c09a3d0 con 0x7f393c095a50 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c06e0d0 con 0x7f393c095e20 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941959700 10 monclient(hunting): authenticate will time out at 1062.190944s 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 0x7f393c12b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 0x7f393c12b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 0x7f393c12b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46510/0 (socket says 172.21.15.104:46510) 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 0x7f393c12b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39202/0 (socket says 172.21.15.104:39202) 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 1 -- 172.21.15.104:0/3892018005 learned_addr learned my addr 172.21.15.104:0/3892018005 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 10 monclient(hunting): get_auth_request con 0x7f393c095e20 auth_method 0 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): get_auth_request con 0x7f393c133da0 auth_method 0 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f393c095a50 0x7f393c12ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 10 monclient(hunting): get_auth_request con 0x7f393c095a50 auth_method 0 2024-09-18T14:25:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 1 -- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 msgr2=0x7f393c12b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 1 -- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 msgr2=0x7f393c12b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 0x7f393c12b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f393bfff700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 0x7f393c12b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3940957700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient(hunting): handle_auth_done global_id 14604 payload 824 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 -- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 msgr2=0x7f393c12b250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 0x7f393c12b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 -- 172.21.15.104:0/3892018005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f393c095a50 msgr2=0x7f393c12ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f393c095a50 0x7f393c12ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient: _finish_hunting 0 2024-09-18T14:25:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 monclient: found mon.noname-a 2024-09-18T14:25:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.195+0000 7f3941158700 1 -- 172.21.15.104:0/3892018005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c00bcf0 con 0x7f393c133da0 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941158700 1 -- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 msgr2=0x7f393c12b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941158700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 0x7f393c12b790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3934006f30 tx=0x7f3934008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 1 -- 172.21.15.104:0/3892018005 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934017070 con 0x7f393c133da0 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: dump: 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: _finish_auth 0 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 5 monclient: authenticate success, global_id 14604 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 10 monclient: _renew_subs 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c06d970 con 0x7f393c133da0 2024-09-18T14:25:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 10 monclient: _renew_subs 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c06ec50 con 0x7f393c133da0 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 1 -- 172.21.15.104:0/3892018005 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3934012800 con 0x7f393c133da0 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 1 -- 172.21.15.104:0/3892018005 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934012bc0 con 0x7f393c133da0 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3938ff9700 10 monclient: dump: 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f393a7fc700 4 set_mon_vals no callback set 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.199+0000 7f3941959700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3941959700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3941959700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3938ff9700 1 -- 172.21.15.104:0/3892018005 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f3934013480 con 0x7f393c133da0 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391c0050b0 con 0x7f393c133da0 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3938ff9700 1 --2- 172.21.15.104:0/3892018005 >> v2:172.21.15.174:6812/17603 conn(0x7f3928035a80 0x7f3928037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3938ff9700 1 -- 172.21.15.104:0/3892018005 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f3934047a30 con 0x7f393c133da0 2024-09-18T14:25:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3940957700 1 --2- 172.21.15.104:0/3892018005 >> v2:172.21.15.174:6812/17603 conn(0x7f3928035a80 0x7f3928037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3940957700 10 monclient: get_auth_request con 0x7f3928035a80 auth_method 0 2024-09-18T14:25:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.203+0000 7f3940957700 1 --2- 172.21.15.104:0/3892018005 >> v2:172.21.15.174:6812/17603 conn(0x7f3928035a80 0x7f3928037f30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3930003a30 tx=0x7f3930007ea0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.211+0000 7f3938ff9700 1 -- 172.21.15.104:0/3892018005 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f39340198c0 con 0x7f393c133da0 2024-09-18T14:25:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.211+0000 7f3938ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.211+0000 7f3938ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.435+0000 7f3941959700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.435+0000 7f3941959700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.435+0000 7f3941959700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.435+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f391c0058e0 con 0x7f393c133da0 2024-09-18T14:25:41.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3938ff9700 1 -- 172.21.15.104:0/3892018005 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f39340192c0 con 0x7f393c133da0 2024-09-18T14:25:41.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3938ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:41.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3938ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:41.445 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:41.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 >> v2:172.21.15.174:6812/17603 conn(0x7f3928035a80 msgr2=0x7f3928037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 --2- 172.21.15.104:0/3892018005 >> v2:172.21.15.174:6812/17603 conn(0x7f3928035a80 0x7f3928037f30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3930003a30 tx=0x7f3930007ea0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 10 monclient: shutdown 2024-09-18T14:25:41.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:41.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 msgr2=0x7f393c12b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 0x7f393c12b790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3934006f30 tx=0x7f3934008040 comp rx=0 tx=0).stop 2024-09-18T14:25:41.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 shutdown_connections 2024-09-18T14:25:41.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 --2- 172.21.15.104:0/3892018005 >> v2:172.21.15.174:6812/17603 conn(0x7f3928035a80 0x7f3928037f30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f393c133da0 0x7f393c12b790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f393c095e20 0x7f393c12b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 --2- 172.21.15.104:0/3892018005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f393c095a50 0x7f393c12ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 >> 172.21.15.104:0/3892018005 conn(0x7f393c0a12c0 msgr2=0x7f393c09d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:41.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 shutdown_connections 2024-09-18T14:25:41.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.439+0000 7f3941959700 1 -- 172.21.15.104:0/3892018005 wait complete. 2024-09-18T14:25:41.458 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T14:25:41.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T14:25:41.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:25:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.555+0000 7f7d315dd700 10 monclient: get_monmap_and_config 2024-09-18T14:25:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.555+0000 7f7d315dd700 10 monclient: build_initial_monmap 2024-09-18T14:25:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.555+0000 7f7d315dd700 10 monclient: monmap: 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:41.563023+0000 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:41.563023+0000 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:41.564 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.555+0000 7f7d315dd700 1 Processor -- start 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 1 -- start start 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient: init 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d2c097c10 addr v1:172.21.15.104:6789/0 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d2c096300 0x7f7d2c09c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient(hunting): picked mon.noname-a con 0x7f7d2c096300 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 0x7f7d2c137770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient(hunting): picked mon.noname-b con 0x7f7d2c135390 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c097fe0 con 0x7f7d2c097c10 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c096700 con 0x7f7d2c096300 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c09cf10 con 0x7f7d2c135390 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 10 monclient(hunting): authenticate will time out at 1062.552535s 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d2c096300 0x7f7d2c09c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 0x7f7d2c137770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7d2c097c10 0x7f7d2c09bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54468/0 (socket says 172.21.15.104:54468) 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d2c096300 0x7f7d2c09c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39204/0 (socket says 172.21.15.104:39204) 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2bfff700 1 -- 172.21.15.104:0/3068947770 learned_addr learned my addr 172.21.15.104:0/3068947770 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:25:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 10 monclient(hunting): get_auth_request con 0x7f7d2c135390 auth_method 0 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): get_auth_request con 0x7f7d2c096300 auth_method 0 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 1 -- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 msgr2=0x7f7d2c137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 1 -- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 msgr2=0x7f7d2c137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 1 --2- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 0x7f7d2c137770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d30ddc700 1 --2- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 0x7f7d2c137770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient(hunting): handle_auth_done global_id 14607 payload 274 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 -- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 msgr2=0x7f7d2c137770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 --2- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 0x7f7d2c137770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 -- 172.21.15.104:0/3068947770 >> v1:172.21.15.104:6789/0 conn(0x7f7d2c097c10 legacy=0x7f7d2c09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 monclient: found mon.noname-a 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 -- 172.21.15.104:0/3068947770 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c095640 con 0x7f7d2c096300 2024-09-18T14:25:41.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 -- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c096300 msgr2=0x7f7d2c09c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2b7fe700 1 --2- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c096300 0x7f7d2c09c2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0098e0 tx=0x7f7d1c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143913a50d8d7317 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2affd700 1 -- 172.21.15.104:0/3068947770 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c009e80 con 0x7f7d2c096300 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2affd700 10 monclient: dump: 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2affd700 10 monclient: _finish_auth 0 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d2affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 5 monclient: authenticate success, global_id 14607 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.559+0000 7f7d315dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2affd700 1 -- 172.21.15.104:0/3068947770 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c00ba70 con 0x7f7d2c096300 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 4 set_mon_vals no callback set 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2affd700 1 -- 172.21.15.104:0/3068947770 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c00bdf0 con 0x7f7d2c096300 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2affd700 10 monclient: dump: 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient: shutdown 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c096300 msgr2=0x7f7d2c09c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 --2- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c096300 0x7f7d2c09c2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0098e0 tx=0x7f7d1c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- 172.21.15.104:0/3068947770 shutdown_connections 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 --2- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c135390 0x7f7d2c137770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 --2- 172.21.15.104:0/3068947770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c096300 0x7f7d2c09c2f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- 172.21.15.104:0/3068947770 >> 172.21.15.104:0/3068947770 conn(0x7f7d2c0a11f0 msgr2=0x7f7d2c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- 172.21.15.104:0/3068947770 shutdown_connections 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- 172.21.15.104:0/3068947770 wait complete. 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient: build_initial_monmap 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient: monmap: 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:41.581 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:41.581 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.581 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:41.581 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 Processor -- start 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- start start 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient: init 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 0x7f7d2c1343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): picked mon.noname-c con 0x7f7d2c097c10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d2c135390 0x7f7d2c1348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): picked mon.noname-b con 0x7f7d2c135390 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 0x7f7d2c08cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): picked mon.noname-a con 0x7f7d2c134e20 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c135250 con 0x7f7d2c134e20 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c06e3e0 con 0x7f7d2c135390 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c090820 con 0x7f7d2c097c10 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d315dd700 10 monclient(hunting): authenticate will time out at 1062.558680s 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 0x7f7d2c08cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 0x7f7d2c08cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39220/0 (socket says 172.21.15.104:39220) 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 -- 172.21.15.104:0/256845117 learned_addr learned my addr 172.21.15.104:0/256845117 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): get_auth_request con 0x7f7d2c134e20 auth_method 0 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.583 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 0x7f7d2c1343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 10 monclient(hunting): get_auth_request con 0x7f7d2c097c10 auth_method 0 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 msgr2=0x7f7d2c1343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 msgr2=0x7f7d2c1343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 0x7f7d2c1343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2bfff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 0x7f7d2c1343a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient(hunting): handle_auth_done global_id 14610 payload 824 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2b7fe700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d2c135390 0x7f7d2c1348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 msgr2=0x7f7d2c1343a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7d2c097c10 0x7f7d2c1343a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d2c135390 msgr2=0x7f7d2c1348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d2c135390 0x7f7d2c1348e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient: _finish_hunting 0 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 monclient: found mon.noname-a 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:41.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d30ddc700 1 -- 172.21.15.104:0/256845117 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c009420 con 0x7f7d2c134e20 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2b7fe700 10 monclient: get_auth_request con 0x7f7d2c135390 auth_method 0 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.563+0000 7f7d2b7fe700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d2c135390 0x7f7d2c1348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d30ddc700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 msgr2=0x7f7d2c08cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d30ddc700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 0x7f7d2c08cc90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d24007f60 tx=0x7f7d24009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24014420 con 0x7f7d2c134e20 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: dump: 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.586 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: _finish_auth 0 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 5 monclient: authenticate success, global_id 14610 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d24014580 con 0x7f7d2c134e20 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 10 monclient: _renew_subs 2024-09-18T14:25:41.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c08f410 con 0x7f7d2c134e20 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24014940 con 0x7f7d2c134e20 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 10 monclient: dump: 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.588 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 10 monclient: _renew_subs 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c090260 con 0x7f7d2c134e20 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2a7fc700 4 set_mon_vals no callback set 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 0x7f7d2c08d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0686b0 con 0x7f7d2c08d1d0 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 0x7f7d2c08d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient: get_auth_request con 0x7f7d2c08d1d0 auth_method 0 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_done global_id 14613 payload 824 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d2bfff700 1 -- 172.21.15.104:0/256845117 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d2c094280 con 0x7f7d2c08d1d0 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f7d2401d410 con 0x7f7d2c134e20 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.567+0000 7f7d13fff700 1 --2- 172.21.15.104:0/256845117 >> v2:172.21.15.174:6812/17603 conn(0x7f7d18035a80 0x7f7d18037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f7d24048650 con 0x7f7d2c134e20 2024-09-18T14:25:41.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d2b7fe700 1 --2- 172.21.15.104:0/256845117 >> v2:172.21.15.174:6812/17603 conn(0x7f7d18035a80 0x7f7d18037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d2b7fe700 10 monclient: get_auth_request con 0x7f7d18035a80 auth_method 0 2024-09-18T14:25:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d2bfff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 msgr2=0x7f7d2c08d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d2bfff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 0x7f7d2c08d5a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e710 tx=0x7f7d20009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20004e10 con 0x7f7d2c08d1d0 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 10 monclient: dump: 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.594 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7d20007170 con 0x7f7d2c08d1d0 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 msgr2=0x7f7d2c08d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 0x7f7d2c08d5a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e710 tx=0x7f7d20009040 comp rx=0 tx=0).stop 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 msgr2=0x7f7d2c08d5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d13fff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08d1d0 0x7f7d2c08d5a0 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e710 tx=0x7f7d20009040 comp rx=0 tx=0).stop 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.571+0000 7f7d2b7fe700 1 --2- 172.21.15.104:0/256845117 >> v2:172.21.15.174:6812/17603 conn(0x7f7d18035a80 0x7f7d18037f30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0966d0 tx=0x7f7d1c016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d315dd700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d315dd700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 0x7f7d2c098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d315dd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c090480 con 0x7f7d2c08dae0 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 0x7f7d2c098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient: get_auth_request con 0x7f7d2c08dae0 auth_method 0 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 10 monclient(hunting): handle_auth_done global_id 14616 payload 824 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.583+0000 7f7d2bfff700 1 -- 172.21.15.104:0/256845117 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7d2c090620 con 0x7f7d2c08dae0 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d2bfff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 msgr2=0x7f7d2c098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d2bfff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 0x7f7d2c098a40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000cfd0 tx=0x7f7d20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20024070 con 0x7f7d2c08dae0 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 10 monclient: dump: 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.597 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f7d200162e0 con 0x7f7d2c08dae0 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 msgr2=0x7f7d2c098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 0x7f7d2c098a40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000cfd0 tx=0x7f7d20017000 comp rx=0 tx=0).stop 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 msgr2=0x7f7d2c098a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d13fff700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c08dae0 0x7f7d2c098a40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:41.598 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 26, 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 0, 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:41.599 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:41.600 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:25:41.601 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:25:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:25:41.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:41.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:41.606 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.607 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:25:41.608 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:25:41.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:25:41.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:25:41.611 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:25:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 >> v2:172.21.15.174:6812/17603 conn(0x7f7d18035a80 msgr2=0x7f7d18037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 --2- 172.21.15.104:0/256845117 >> v2:172.21.15.174:6812/17603 conn(0x7f7d18035a80 0x7f7d18037f30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0966d0 tx=0x7f7d1c016000 comp rx=0 tx=0).stop 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 10 monclient: shutdown 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 msgr2=0x7f7d2c08cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d30ddc700 1 -- 172.21.15.104:0/256845117 reap_dead start 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 0x7f7d2c08cc90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d24007f60 tx=0x7f7d24009910 comp rx=0 tx=0).stop 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 shutdown_connections 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 --2- 172.21.15.104:0/256845117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d2c134e20 0x7f7d2c08cc90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 >> 172.21.15.104:0/256845117 conn(0x7f7d2c0a11f0 msgr2=0x7f7d2c094710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 shutdown_connections 2024-09-18T14:25:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.587+0000 7f7d315dd700 1 -- 172.21.15.104:0/256845117 wait complete. 2024-09-18T14:25:41.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:25:41.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T14:25:41.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient: get_monmap_and_config 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient: build_initial_monmap 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient: monmap: 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:41.724631+0000 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:41.724631+0000 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.728 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 1 Processor -- start 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 1 -- start start 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient: init 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7828097c10 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7828096300 addr v1:172.21.15.104:6789/0 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f78281363a0 0x7f7828138780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient(hunting): picked mon.noname-a con 0x7f78281363a0 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782809ca00 con 0x7f7828096300 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7828097fe0 con 0x7f7828097c10 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7828096700 con 0x7f78281363a0 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782d63b700 10 monclient(hunting): authenticate will time out at 1062.714171s 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f78281363a0 0x7f7828138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f78277fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7828096300 0x7f782809c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54470/0 (socket says 172.21.15.104:54470) 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): get_auth_request con 0x7f78281363a0 auth_method 0 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f78277fe700 1 -- 172.21.15.104:0/1666483221 learned_addr learned my addr 172.21.15.104:0/1666483221 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient(hunting): handle_auth_done global_id 14619 payload 274 2024-09-18T14:25:41.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 1 -- 172.21.15.104:0/1666483221 >> v1:172.21.15.174:6789/0 conn(0x7f7828097c10 legacy=0x7f782809bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:41.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 1 -- 172.21.15.104:0/1666483221 >> v1:172.21.15.104:6789/0 conn(0x7f7828096300 legacy=0x7f782809c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:41.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient: _finish_hunting 0 2024-09-18T14:25:41.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 1 monclient: found mon.noname-a 2024-09-18T14:25:41.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:41.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.719+0000 7f782ce3a700 1 -- 172.21.15.104:0/1666483221 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782809cf10 con 0x7f78281363a0 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782ce3a700 1 -- 172.21.15.104:0/1666483221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 msgr2=0x7f7828138780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782ce3a700 1 --2- 172.21.15.104:0/1666483221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f7828138780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f78180098e0 tx=0x7f78180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ffba0bb82250e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 1 -- 172.21.15.104:0/1666483221 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781800ba70 con 0x7f78281363a0 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: dump: 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.736 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: _finish_auth 0 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 5 monclient: authenticate success, global_id 14619 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 1 -- 172.21.15.104:0/1666483221 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781800bbd0 con 0x7f78281363a0 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:41.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 4 set_mon_vals no callback set 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 1 -- 172.21.15.104:0/1666483221 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818011420 con 0x7f78281363a0 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f7826ffd700 10 monclient: dump: 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.738 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 10 monclient: shutdown 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 1 -- 172.21.15.104:0/1666483221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 msgr2=0x7f7828138780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 1 --2- 172.21.15.104:0/1666483221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f7828138780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f78180098e0 tx=0x7f78180091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 1 -- 172.21.15.104:0/1666483221 shutdown_connections 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 1 --2- 172.21.15.104:0/1666483221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f7828138780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 1 -- 172.21.15.104:0/1666483221 >> 172.21.15.104:0/1666483221 conn(0x7f78280a11f0 msgr2=0x7f78280a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 1 -- 172.21.15.104:0/1666483221 shutdown_connections 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 1 -- 172.21.15.104:0/1666483221 wait complete. 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 10 monclient: build_initial_monmap 2024-09-18T14:25:41.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.723+0000 7f782d63b700 10 monclient: monmap: 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 Processor -- start 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 -- start start 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient: init 2024-09-18T14:25:41.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7828096300 0x7f7828094c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): picked mon.noname-b con 0x7f7828096300 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 0x7f78281353b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): picked mon.noname-c con 0x7f7828097c10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f78281358f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): picked mon.noname-a con 0x7f78281363a0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782809d4b0 con 0x7f78281363a0 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782808d730 con 0x7f7828096300 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782808efc0 con 0x7f7828097c10 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782d63b700 10 monclient(hunting): authenticate will time out at 1062.721764s 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 0x7f78281353b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 0x7f78281353b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46532/0 (socket says 172.21.15.104:46532) 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 1 -- 172.21.15.104:0/1867249493 learned_addr learned my addr 172.21.15.104:0/1867249493 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 10 monclient(hunting): get_auth_request con 0x7f7828097c10 auth_method 0 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f78281358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 msgr2=0x7f78281353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 msgr2=0x7f78281353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:41.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 0x7f78281353b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): get_auth_request con 0x7f78281363a0 auth_method 0 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f78277fe700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 0x7f78281353b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7828096300 0x7f7828094c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 10 monclient(hunting): get_auth_request con 0x7f7828096300 auth_method 0 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient(hunting): handle_auth_done global_id 14622 payload 824 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 msgr2=0x7f78281353b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828097c10 0x7f78281353b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7828096300 msgr2=0x7f7828094c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7828096300 0x7f7828094c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient: _finish_hunting 0 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 1 monclient: found mon.noname-a 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.727+0000 7f782ce3a700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818009420 con 0x7f78281363a0 2024-09-18T14:25:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782ce3a700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 msgr2=0x7f78281358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782ce3a700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f78281358f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7818015000 tx=0x7f7818016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781801b890 con 0x7f78281363a0 2024-09-18T14:25:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: dump: 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: _finish_auth 0 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 5 monclient: authenticate success, global_id 14622 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781801b9f0 con 0x7f78281363a0 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 10 monclient: _renew_subs 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782809fce0 con 0x7f78281363a0 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781801bd70 con 0x7f78281363a0 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f78267fc700 4 set_mon_vals no callback set 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 10 monclient: dump: 2024-09-18T14:25:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 10 monclient: _renew_subs 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782808ed90 con 0x7f78281363a0 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78280686b0 con 0x7f7828135e30 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f7828135e30 auth_method 0 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f781801c660 con 0x7f78281363a0 2024-09-18T14:25:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 1 --2- 172.21.15.104:0/1867249493 >> v2:172.21.15.174:6812/17603 conn(0x7f7814035a80 0x7f7814037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f7818049e50 con 0x7f78281363a0 2024-09-18T14:25:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f78277fe700 1 --2- 172.21.15.104:0/1867249493 >> v2:172.21.15.174:6812/17603 conn(0x7f7814035a80 0x7f7814037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.731+0000 7f78277fe700 10 monclient: get_auth_request con 0x7f7814035a80 auth_method 0 2024-09-18T14:25:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.735+0000 7f78277fe700 1 --2- 172.21.15.104:0/1867249493 >> v2:172.21.15.174:6812/17603 conn(0x7f7814035a80 0x7f7814037f30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7820009610 tx=0x7f78200096d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:41.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f7828135e30 auth_method 0 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:41.935+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:25:42.208 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:42.208 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21162]: starting. 2024-09-18T14:25:42.208 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21162]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21127 2024-09-18T14:25:42.209 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21162]: created object 0... 2024-09-18T14:25:42.209 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21162]: created object 25... 2024-09-18T14:25:42.209 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21162]: created object 49... 2024-09-18T14:25:42.209 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21162]: finishing. 2024-09-18T14:25:42.209 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21162]: shutting down. 2024-09-18T14:25:42.269 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:42.270 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21164]: starting. 2024-09-18T14:25:42.270 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21164]: listing objects. 2024-09-18T14:25:42.270 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21164]: listed object 0... 2024-09-18T14:25:42.270 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21164]: listed object 25... 2024-09-18T14:25:42.270 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21164]: saw 50 objects 2024-09-18T14:25:42.270 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21164]: shutting down. 2024-09-18T14:25:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f7828135e30 auth_method 0 2024-09-18T14:25:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:42.339+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:25:42.798 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:42.798 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21163]: starting. 2024-09-18T14:25:42.799 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21163]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21127 2024-09-18T14:25:42.799 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21163]: shutting down. 2024-09-18T14:25:42.799 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:42.799 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:42.800 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:42.801 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f7828135e30 auth_method 0 2024-09-18T14:25:43.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:43.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:43.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:43.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:43.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:43.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:43.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:43.143+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:25:44.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.731+0000 7f780effd700 10 monclient: tick 2024-09-18T14:25:44.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.731+0000 7f780effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:44.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.731+0000 7f780effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:44.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f7828135e30 auth_method 0 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:44.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:44.747+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-18T14:25:44.982 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21215]: starting. 2024-09-18T14:25:44.982 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21215]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21214 2024-09-18T14:25:44.982 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21215]: created object 0... 2024-09-18T14:25:44.982 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21215]: created object 25... 2024-09-18T14:25:44.982 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21215]: created object 49... 2024-09-18T14:25:44.983 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21215]: finishing. 2024-09-18T14:25:44.983 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21215]: shutting down. 2024-09-18T14:25:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 10 monclient: tick 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 5 monclient: _check_tell_commands timeout tell command 1 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 msgr2=0x7f782808b700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7828135e30 0x7f782808b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f780effd700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7800006110 con 0x7f78000035e0 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f78000035e0 auth_method 0 2024-09-18T14:25:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 msgr2=0x7f7800005a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:25:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 msgr2=0x7f7800005a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:25:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:25:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.731+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:45.818 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21216]: starting. 2024-09-18T14:25:45.819 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21216]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21214 2024-09-18T14:25:45.819 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21216]: shutting down. 2024-09-18T14:25:45.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f78000035e0 auth_method 0 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 10 monclient(hunting): handle_auth_done global_id 24101 payload 824 2024-09-18T14:25:45.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7800006280 con 0x7f78000035e0 2024-09-18T14:25:45.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 msgr2=0x7f7800005a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:45.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f781c019ad0 tx=0x7f781c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.935+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c01e070 con 0x7f78000035e0 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 10 monclient: dump: 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:45.944 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f781c022410 con 0x7f78000035e0 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:45.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:45.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 msgr2=0x7f7800005a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:45.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f781c019ad0 tx=0x7f781c009040 comp rx=0 tx=0).stop 2024-09-18T14:25:45.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 msgr2=0x7f7800005a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:45.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.939+0000 7f780ffff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f78000035e0 0x7f7800005a60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:45.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f782d63b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f782d63b700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 0x7f782808c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f782d63b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782808c580 con 0x7f782808bc40 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 0x7f782808c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient: get_auth_request con 0x7f782808bc40 auth_method 0 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:45.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:45.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 10 monclient(hunting): handle_auth_done global_id 24104 payload 824 2024-09-18T14:25:45.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.955+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7828063450 con 0x7f782808bc40 2024-09-18T14:25:45.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f7827fff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 msgr2=0x7f782808c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:45.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f7827fff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 0x7f782808c040 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f781c003400 tx=0x7f781c024000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:45.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c033070 con 0x7f782808bc40 2024-09-18T14:25:45.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:45.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:25:45.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 10 monclient: dump: 2024-09-18T14:25:45.966 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:45.966 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f781c017ce0 con 0x7f782808bc40 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:25:45.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 msgr2=0x7f782808c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 0x7f782808c040 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f781c003400 tx=0x7f781c024000 comp rx=0 tx=0).stop 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 msgr2=0x7f782808c040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782ce3a700 1 -- 172.21.15.104:0/1867249493 reap_dead start 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f780ffff700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f782808bc40 0x7f782808c040 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:45.968 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 26, 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 4, 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:25:45.969 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:45.970 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:45.971 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:45.973 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.974 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.975 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.976 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:45.977 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:45.979 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:45.980 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:45.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.981 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:25:45.981 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:25:45.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.981 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:45.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:45.983 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:25:45.983 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:25:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 >> v2:172.21.15.174:6812/17603 conn(0x7f7814035a80 msgr2=0x7f7814037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 --2- 172.21.15.104:0/1867249493 >> v2:172.21.15.174:6812/17603 conn(0x7f7814035a80 0x7f7814037f30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7820009610 tx=0x7f78200096d0 comp rx=0 tx=0).stop 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 10 monclient: shutdown 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 msgr2=0x7f78281358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f78281358f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7818015000 tx=0x7f7818016000 comp rx=0 tx=0).stop 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 shutdown_connections 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 --2- 172.21.15.104:0/1867249493 >> v2:172.21.15.174:6812/17603 conn(0x7f7814035a80 0x7f7814037f30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 --2- 172.21.15.104:0/1867249493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78281363a0 0x7f78281358f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 >> 172.21.15.104:0/1867249493 conn(0x7f78280a11f0 msgr2=0x7f782806d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 shutdown_connections 2024-09-18T14:25:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:45.959+0000 7f782d63b700 1 -- 172.21.15.104:0/1867249493 wait complete. 2024-09-18T14:25:45.986 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:25:45.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T14:25:45.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient: get_monmap_and_config 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient: build_initial_monmap 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient: monmap: 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:46.100387+0000 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:46.100387+0000 2024-09-18T14:25:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:46.102 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 1 Processor -- start 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 1 -- start start 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient: init 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bb809e830 addr v1:172.21.15.104:6789/0 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9bb809ec00 0x7f9bb809f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient(hunting): picked mon.noname-c con 0x7f9bb809ec00 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb8136e60 0x7f9bb8139240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient(hunting): picked mon.noname-b con 0x7f9bb8136e60 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb809b7a0 con 0x7f9bb809e830 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb8099f10 con 0x7f9bb809ec00 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb809b4c0 con 0x7f9bb8136e60 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbe46b700 10 monclient(hunting): authenticate will time out at 1067.090594s 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb8136e60 0x7f9bb8139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbd469700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9bb809e830 0x7f9bb8131d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54486/0 (socket says 172.21.15.104:54486) 2024-09-18T14:25:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbd469700 1 -- 172.21.15.104:0/2157340076 learned_addr learned my addr 172.21.15.104:0/2157340076 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): get_auth_request con 0x7f9bb8136e60 auth_method 0 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 1 --2- 172.21.15.104:0/2157340076 >> v2:172.21.15.174:3300/0 conn(0x7f9bb809ec00 0x7f9bb809f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 10 monclient(hunting): get_auth_request con 0x7f9bb809ec00 auth_method 0 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbcc68700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_done global_id 24107 payload 274 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/2157340076 >> v2:172.21.15.174:3300/0 conn(0x7f9bb809ec00 msgr2=0x7f9bb809f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 1 --2- 172.21.15.104:0/2157340076 >> v2:172.21.15.174:3300/0 conn(0x7f9bb809ec00 0x7f9bb809f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/2157340076 >> v1:172.21.15.104:6789/0 conn(0x7f9bb809e830 legacy=0x7f9bb8131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient: _finish_hunting 0 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 1 monclient: found mon.noname-b 2024-09-18T14:25:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:25:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.095+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/2157340076 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb809ba80 con 0x7f9bb8136e60 2024-09-18T14:25:46.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/2157340076 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb8136e60 msgr2=0x7f9bb8139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:46.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbdc6a700 1 --2- 172.21.15.104:0/2157340076 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb8136e60 0x7f9bb8139240 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40098e0 tx=0x7f9bb40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edbbce1652c2dbff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 1 -- 172.21.15.104:0/2157340076 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4011840 con 0x7f9bb8136e60 2024-09-18T14:25:46.115 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: dump: 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: _finish_auth 0 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 5 monclient: authenticate success, global_id 24107 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 1 -- 172.21.15.104:0/2157340076 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb40119a0 con 0x7f9bb8136e60 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 4 set_mon_vals no callback set 2024-09-18T14:25:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 1 -- 172.21.15.104:0/2157340076 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4011d40 con 0x7f9bb8136e60 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9baffff700 10 monclient: dump: 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.118 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 10 monclient: shutdown 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 -- 172.21.15.104:0/2157340076 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb8136e60 msgr2=0x7f9bb8139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/2157340076 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb8136e60 0x7f9bb8139240 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40098e0 tx=0x7f9bb40091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 -- 172.21.15.104:0/2157340076 shutdown_connections 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/2157340076 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb8136e60 0x7f9bb8139240 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/2157340076 >> v2:172.21.15.174:3300/0 conn(0x7f9bb809ec00 0x7f9bb809f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 -- 172.21.15.104:0/2157340076 >> 172.21.15.104:0/2157340076 conn(0x7f9bb80676a0 msgr2=0x7f9bb8067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 -- 172.21.15.104:0/2157340076 shutdown_connections 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.099+0000 7f9bbe46b700 1 -- 172.21.15.104:0/2157340076 wait complete. 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient: build_initial_monmap 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient: monmap: 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.120 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 Processor -- start 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 -- start start 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient: init 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb809e830 0x7f9bb8136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): picked mon.noname-b con 0x7f9bb809e830 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb809ec00 0x7f9bb812f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): picked mon.noname-c con 0x7f9bb809ec00 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 0x7f9bb812f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): picked mon.noname-a con 0x7f9bb8136e60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb809f5e0 con 0x7f9bb8136e60 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8130b30 con 0x7f9bb809e830 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8133430 con 0x7f9bb809ec00 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:46.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbe46b700 10 monclient(hunting): authenticate will time out at 1067.098190s 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 0x7f9bb812f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb809ec00 0x7f9bb812f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb809e830 0x7f9bb8136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 0x7f9bb812f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39288/0 (socket says 172.21.15.104:39288) 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb809ec00 0x7f9bb812f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46650/0 (socket says 172.21.15.104:46650) 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/542187590 learned_addr learned my addr 172.21.15.104:0/542187590 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): get_auth_request con 0x7f9bb8136e60 auth_method 0 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 10 monclient(hunting): get_auth_request con 0x7f9bb809e830 auth_method 0 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 10 monclient(hunting): get_auth_request con 0x7f9bb809ec00 auth_method 0 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbcc68700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient(hunting): handle_auth_done global_id 14652 payload 824 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb809ec00 msgr2=0x7f9bb812f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9bb809ec00 0x7f9bb812f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb809e830 msgr2=0x7f9bb8136380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb809e830 0x7f9bb8136380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient: _finish_hunting 0 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 monclient: found mon.noname-a 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/542187590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb4009420 con 0x7f9bb8136e60 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.103+0000 7f9bbd469700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb809e830 0x7f9bb8136380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 msgr2=0x7f9bb812f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbdc6a700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 0x7f9bb812f870 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40091e0 tx=0x7f9bb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4024440 con 0x7f9bb8136e60 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: dump: 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.127 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: _finish_auth 0 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 5 monclient: authenticate success, global_id 14652 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb40245a0 con 0x7f9bb8136e60 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 10 monclient: _renew_subs 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb8131410 con 0x7f9bb8136e60 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4024960 con 0x7f9bb8136e60 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bad7fa700 10 monclient: dump: 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9baf7fe700 4 set_mon_vals no callback set 2024-09-18T14:25:46.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 10 monclient: _renew_subs 2024-09-18T14:25:46.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb81340f0 con 0x7f9bb8136e60 2024-09-18T14:25:46.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:46.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 0x7f9bb80a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8064840 con 0x7f9bb812fdb0 2024-09-18T14:25:46.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbd469700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 0x7f9bb80a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbd469700 10 monclient: get_auth_request con 0x7f9bb812fdb0 auth_method 0 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbd469700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbd469700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.107+0000 7f9bbd469700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f9bb4024d50 con 0x7f9bb8136e60 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bad7fa700 1 --2- 172.21.15.104:0/542187590 >> v2:172.21.15.174:6812/17603 conn(0x7f9b9c035a80 0x7f9b9c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbd469700 10 monclient(hunting): handle_auth_done global_id 5347 payload 824 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9bb4049990 con 0x7f9bb8136e60 2024-09-18T14:25:46.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbd469700 1 -- 172.21.15.104:0/542187590 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bb8130180 con 0x7f9bb812fdb0 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbcc68700 1 --2- 172.21.15.104:0/542187590 >> v2:172.21.15.174:6812/17603 conn(0x7f9b9c035a80 0x7f9b9c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbcc68700 10 monclient: get_auth_request con 0x7f9b9c035a80 auth_method 0 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbd469700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 msgr2=0x7f9bb80a3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbd469700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 0x7f9bb80a3d50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d3b0 tx=0x7f9bb0009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0007530 con 0x7f9bb812fdb0 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bad7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:46.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bad7fa700 10 monclient: dump: 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.111+0000 7f9bbcc68700 1 --2- 172.21.15.104:0/542187590 >> v2:172.21.15.174:6812/17603 conn(0x7f9b9c035a80 0x7f9b9c037f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009db0 tx=0x7f9ba80091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.115+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9bb0007690 con 0x7f9bb812fdb0 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.115+0000 7f9bad7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.115+0000 7f9bad7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.115+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 msgr2=0x7f9bb80a3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.115+0000 7f9bad7fa700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 0x7f9bb80a3d50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d3b0 tx=0x7f9bb0009040 comp rx=0 tx=0).stop 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.115+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 msgr2=0x7f9bb80a3d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.115+0000 7f9bad7fa700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb812fdb0 0x7f9bb80a3d50 secure :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d3b0 tx=0x7f9bb0009040 comp rx=0 tx=0).stop 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.123+0000 7f9bbe46b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.123+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 0x7f9bb80a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.123+0000 7f9bbe46b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.123+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8063450 con 0x7f9bb80a4290 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 0x7f9bb80a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient: get_auth_request con 0x7f9bb80a4290 auth_method 0 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 10 monclient(hunting): handle_auth_done global_id 5350 payload 824 2024-09-18T14:25:46.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 1 -- 172.21.15.104:0/542187590 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9bb80689c0 con 0x7f9bb80a4290 2024-09-18T14:25:46.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 msgr2=0x7f9bb80a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:46.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bbd469700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 0x7f9bb80a4690 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d3b0 tx=0x7f9bb0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0023810 con 0x7f9bb80a4290 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bad7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.127+0000 7f9bad7fa700 10 monclient: dump: 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.140 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f9bb0023970 con 0x7f9bb80a4290 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bad7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bad7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 msgr2=0x7f9bb80a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bad7fa700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 0x7f9bb80a4690 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d3b0 tx=0x7f9bb0017000 comp rx=0 tx=0).stop 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bad7fa700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 msgr2=0x7f9bb80a4690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:46.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bad7fa700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bb80a4290 0x7f9bb80a4690 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 26, 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:25:46.142 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 5, 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:46.143 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:46.144 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:25:46.145 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:46.146 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.147 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:25:46.148 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:46.149 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:25:46.150 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:25:46.151 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:46.152 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:25:46.153 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:46.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.155 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 7 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:46.156 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:46.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:46.157 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:25:46.157 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:25:46.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 >> v2:172.21.15.174:6812/17603 conn(0x7f9b9c035a80 msgr2=0x7f9b9c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/542187590 >> v2:172.21.15.174:6812/17603 conn(0x7f9b9c035a80 0x7f9b9c037f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009db0 tx=0x7f9ba80091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 10 monclient: shutdown 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 msgr2=0x7f9bb812f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbdc6a700 1 -- 172.21.15.104:0/542187590 reap_dead start 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 0x7f9bb812f870 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40091e0 tx=0x7f9bb4017000 comp rx=0 tx=0).stop 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 shutdown_connections 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 --2- 172.21.15.104:0/542187590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bb8136e60 0x7f9bb812f870 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 >> 172.21.15.104:0/542187590 conn(0x7f9bb80676a0 msgr2=0x7f9bb809d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 shutdown_connections 2024-09-18T14:25:46.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.131+0000 7f9bbe46b700 1 -- 172.21.15.104:0/542187590 wait complete. 2024-09-18T14:25:46.160 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:25:46.160 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T14:25:46.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient: get_monmap_and_config 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient: build_initial_monmap 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient: monmap: 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:46.273837+0000 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:46.273837+0000 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 1 Processor -- start 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 1 -- start start 2024-09-18T14:25:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient: init 2024-09-18T14:25:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T14:25:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3828095a50 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3828095e20 addr v1:172.21.15.104:6789/0 2024-09-18T14:25:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3828134db0 0x7f3828137190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient(hunting): picked mon.noname-a con 0x7f3828134db0 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3828096bf0 con 0x7f3828095e20 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382812f510 con 0x7f3828095a50 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3828097030 con 0x7f3828134db0 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382f13b700 10 monclient(hunting): authenticate will time out at 1067.263438s 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3828134db0 0x7f3828137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3828134db0 0x7f3828137190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39292/0 (socket says 172.21.15.104:39292) 2024-09-18T14:25:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 1 -- 172.21.15.104:0/3405184457 learned_addr learned my addr 172.21.15.104:0/3405184457 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 10 monclient(hunting): get_auth_request con 0x7f3828134db0 auth_method 0 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.267+0000 7f382e93a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 10 monclient(hunting): handle_auth_done global_id 14655 payload 274 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 1 -- 172.21.15.104:0/3405184457 >> v1:172.21.15.174:6789/0 conn(0x7f3828095a50 legacy=0x7f382812ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 1 -- 172.21.15.104:0/3405184457 >> v1:172.21.15.104:6789/0 conn(0x7f3828095e20 legacy=0x7f3828096240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 10 monclient: _finish_hunting 0 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 1 monclient: found mon.noname-a 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 1 -- 172.21.15.104:0/3405184457 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828098530 con 0x7f3828134db0 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 1 -- 172.21.15.104:0/3405184457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828134db0 msgr2=0x7f3828137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382e93a700 1 --2- 172.21.15.104:0/3405184457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828134db0 0x7f3828137190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3820009f20 tx=0x7f3820009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b97d014095ebc83 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 1 -- 172.21.15.104:0/3405184457 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382000e070 con 0x7f3828134db0 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: dump: 2024-09-18T14:25:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: _finish_auth 0 2024-09-18T14:25:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 1 -- 172.21.15.104:0/3405184457 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382000bc90 con 0x7f3828134db0 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382f13b700 5 monclient: authenticate success, global_id 14655 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382f13b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382f13b700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 1 -- 172.21.15.104:0/3405184457 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820012660 con 0x7f3828134db0 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382f13b700 4 set_mon_vals no callback set 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.271+0000 7f382d137700 10 monclient: dump: 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.285 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient: shutdown 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- 172.21.15.104:0/3405184457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828134db0 msgr2=0x7f3828137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 --2- 172.21.15.104:0/3405184457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828134db0 0x7f3828137190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3820009f20 tx=0x7f3820009840 comp rx=0 tx=0).stop 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- 172.21.15.104:0/3405184457 shutdown_connections 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 --2- 172.21.15.104:0/3405184457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828134db0 0x7f3828137190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- 172.21.15.104:0/3405184457 >> 172.21.15.104:0/3405184457 conn(0x7f38280a12c0 msgr2=0x7f38280a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- 172.21.15.104:0/3405184457 shutdown_connections 2024-09-18T14:25:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- 172.21.15.104:0/3405184457 wait complete. 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient: build_initial_monmap 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient: monmap: 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 Processor -- start 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- start start 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient: init 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 0x7f3828130220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): picked mon.noname-a con 0x7f3828095a50 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3828095e20 0x7f3828133a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): picked mon.noname-c con 0x7f3828095e20 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3828134db0 0x7f382806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): picked mon.noname-b con 0x7f3828134db0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382809fd20 con 0x7f3828095a50 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828131540 con 0x7f3828134db0 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3828130880 con 0x7f3828095e20 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382f13b700 10 monclient(hunting): authenticate will time out at 1067.271605s 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382d938700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3828095e20 0x7f3828133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382e139700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 0x7f3828130220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382d938700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3828095e20 0x7f3828133a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46662/0 (socket says 172.21.15.104:46662) 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382e139700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 0x7f3828130220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39304/0 (socket says 172.21.15.104:39304) 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382d938700 1 -- 172.21.15.104:0/925403187 learned_addr learned my addr 172.21.15.104:0/925403187 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382d938700 10 monclient(hunting): get_auth_request con 0x7f3828095e20 auth_method 0 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382d938700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382d938700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382d938700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382e139700 10 monclient(hunting): get_auth_request con 0x7f3828095a50 auth_method 0 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382e139700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382e139700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.275+0000 7f382e139700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e93a700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3828134db0 0x7f382806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382d938700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382d938700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382d938700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e93a700 10 monclient(hunting): get_auth_request con 0x7f3828134db0 auth_method 0 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e93a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e93a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e93a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 10 monclient(hunting): handle_auth_done global_id 14658 payload 824 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 -- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3828095e20 msgr2=0x7f3828133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3828095e20 0x7f3828133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 -- 172.21.15.104:0/925403187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3828134db0 msgr2=0x7f382806d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3828134db0 0x7f382806d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 10 monclient: _finish_hunting 0 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 monclient: found mon.noname-a 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 -- 172.21.15.104:0/925403187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820009ae0 con 0x7f3828095a50 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e93a700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3828134db0 0x7f382806d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 -- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 msgr2=0x7f3828130220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382e139700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 0x7f3828130220 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f381800c4e0 tx=0x7f3818007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 1 -- 172.21.15.104:0/925403187 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38180117a0 con 0x7f3828095a50 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: dump: 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.299 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: _finish_auth 0 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382f13b700 5 monclient: authenticate success, global_id 14658 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 1 -- 172.21.15.104:0/925403187 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3818011920 con 0x7f3828095a50 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 1 -- 172.21.15.104:0/925403187 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818011ce0 con 0x7f3828095a50 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f381e7fc700 10 monclient: dump: 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382f13b700 10 monclient: _renew_subs 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382c936700 4 set_mon_vals no callback set 2024-09-18T14:25:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382f13b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3828130b60 con 0x7f3828095a50 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382f13b700 10 monclient: _renew_subs 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382f13b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.279+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382806f340 con 0x7f3828095a50 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f382f13b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f382f13b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f382f13b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38280633b0 con 0x7f3828095a50 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f381e7fc700 1 -- 172.21.15.104:0/925403187 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f3818012630 con 0x7f3828095a50 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f381e7fc700 1 --2- 172.21.15.104:0/925403187 >> v2:172.21.15.174:6812/17603 conn(0x7f380c035a80 0x7f380c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f381e7fc700 1 -- 172.21.15.104:0/925403187 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3818047b60 con 0x7f3828095a50 2024-09-18T14:25:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f382d938700 1 --2- 172.21.15.104:0/925403187 >> v2:172.21.15.174:6812/17603 conn(0x7f380c035a80 0x7f380c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.283+0000 7f382d938700 10 monclient: get_auth_request con 0x7f380c035a80 auth_method 0 2024-09-18T14:25:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.287+0000 7f382d938700 1 --2- 172.21.15.104:0/925403187 >> v2:172.21.15.174:6812/17603 conn(0x7f380c035a80 0x7f380c037f30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f382806f6c0 tx=0x7f3824007f60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.291+0000 7f381e7fc700 1 -- 172.21.15.104:0/925403187 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3818011e40 con 0x7f3828095a50 2024-09-18T14:25:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.291+0000 7f381e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.291+0000 7f381e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.511+0000 7f382f13b700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T14:25:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.511+0000 7f382f13b700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T14:25:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.511+0000 7f382f13b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.511+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3828130f00 con 0x7f3828095a50 2024-09-18T14:25:46.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.515+0000 7f381e7fc700 1 -- 172.21.15.104:0/925403187 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f38180102e0 con 0x7f3828095a50 2024-09-18T14:25:46.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.515+0000 7f381e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T14:25:46.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.515+0000 7f381e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:46.521 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.521 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=11,kv=8,logm=70} crc {auth=3552602209,config=3140514009,health=1656108224,kv=1715430131,logm=2827396108}) 2024-09-18T14:25:46.521 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=7,config=2,health=12,kv=8,logm=71} crc {auth=217663830,config=3140514009,health=3466542341,kv=1394062618,logm=614999810}) 2024-09-18T14:25:46.521 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.521 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=11,kv=8,logm=70} crc {auth=3552602209,config=3140514009,health=1656108224,kv=1715430131,logm=2827396108}) 2024-09-18T14:25:46.521 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=1,health=11,kv=7,logm=73} crc {auth=1961312364,config=1228700967,health=1131072225,kv=268805236,logm=568766050}) 2024-09-18T14:25:46.523 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.523 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891257476}) 2024-09-18T14:25:46.524 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=702626495}) 2024-09-18T14:25:46.524 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.524 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891257476}) 2024-09-18T14:25:46.524 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.515+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2545417484}) 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 >> v2:172.21.15.174:6812/17603 conn(0x7f380c035a80 msgr2=0x7f380c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 --2- 172.21.15.104:0/925403187 >> v2:172.21.15.174:6812/17603 conn(0x7f380c035a80 0x7f380c037f30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f382806f6c0 tx=0x7f3824007f60 comp rx=0 tx=0).stop 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 10 monclient: shutdown 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 msgr2=0x7f3828130220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 0x7f3828130220 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f381800c4e0 tx=0x7f3818007ea0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 shutdown_connections 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 --2- 172.21.15.104:0/925403187 >> v2:172.21.15.174:6812/17603 conn(0x7f380c035a80 0x7f380c037f30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3828134db0 0x7f382806d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f3828095e20 0x7f3828133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 --2- 172.21.15.104:0/925403187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3828095a50 0x7f3828130220 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 >> 172.21.15.104:0/925403187 conn(0x7f38280a12c0 msgr2=0x7f382809abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 shutdown_connections 2024-09-18T14:25:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:46.519+0000 7f382f13b700 1 -- 172.21.15.104:0/925403187 wait complete. 2024-09-18T14:25:46.526 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.519+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.526 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.519+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=877003825}) 2024-09-18T14:25:46.527 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.519+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=966122580}) 2024-09-18T14:25:46.527 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.519+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.527 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.519+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=877003825}) 2024-09-18T14:25:46.527 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.519+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=367792267}) 2024-09-18T14:25:46.528 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.528 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793857624}) 2024-09-18T14:25:46.528 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691687080}) 2024-09-18T14:25:46.528 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.528 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793857624}) 2024-09-18T14:25:46.528 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=570538624}) 2024-09-18T14:25:46.530 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.530 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220147569}) 2024-09-18T14:25:46.530 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173309229}) 2024-09-18T14:25:46.530 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.530 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220147569}) 2024-09-18T14:25:46.531 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.523+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125881180}) 2024-09-18T14:25:46.532 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.532 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071945445}) 2024-09-18T14:25:46.532 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1997212455}) 2024-09-18T14:25:46.532 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.532 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071945445}) 2024-09-18T14:25:46.532 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945946467}) 2024-09-18T14:25:46.534 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.534 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497027866}) 2024-09-18T14:25:46.534 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3393794324}) 2024-09-18T14:25:46.534 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.534 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497027866}) 2024-09-18T14:25:46.534 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1472043068}) 2024-09-18T14:25:46.535 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.535 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459685514}) 2024-09-18T14:25:46.535 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448610622}) 2024-09-18T14:25:46.535 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.535 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459685514}) 2024-09-18T14:25:46.536 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.527+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2829754570}) 2024-09-18T14:25:46.537 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.537 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173482712}) 2024-09-18T14:25:46.537 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481311868}) 2024-09-18T14:25:46.537 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.537 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173482712}) 2024-09-18T14:25:46.537 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2625409591}) 2024-09-18T14:25:46.539 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T14:25:46.539 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.540 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=1544734353,mdsmap=1531151339,mgr=1292546760,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=275518038}) 2024-09-18T14:25:46.540 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=3195598224,mdsmap=1531151339,mgr=3617533101,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=275518038}) 2024-09-18T14:25:46.540 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.540 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=1544734353,mdsmap=1531151339,mgr=1292546760,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=275518038}) 2024-09-18T14:25:46.540 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.531+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5} crc {logm=819555548,mdsmap=1531151339,mgr=3617533101,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=275518038,mon_config_key=1784637070}) 2024-09-18T14:25:46.541 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.541 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=85} crc {mon_config_key=178189784,monmap=530371891,osd_metadata=250104756,osd_pg_creating=2491259426,osd_snap=3178633393,osdmap=1411355360}) 2024-09-18T14:25:46.541 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=84} crc {mon_config_key=1784637070,monmap=530371891,osd_metadata=2265752427,osd_pg_creating=2491259426,osd_snap=956710772,osdmap=2383148670}) 2024-09-18T14:25:46.541 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.541 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=85} crc {mon_config_key=178189784,monmap=530371891,osd_metadata=250104756,osd_pg_creating=2491259426,osd_snap=3178633393,osdmap=1411355360}) 2024-09-18T14:25:46.541 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=84} crc {mon_config_key=1784637070,monmap=3596764712,osd_metadata=2265752427,osd_pg_creating=2491259426,osd_snap=3178633393,osdmap=3769693464}) 2024-09-18T14:25:46.542 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.542 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3768167486}) 2024-09-18T14:25:46.542 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4204909808}) 2024-09-18T14:25:46.542 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.542 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3768167486}) 2024-09-18T14:25:46.542 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=58262766}) 2024-09-18T14:25:46.544 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.544 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2605566472}) 2024-09-18T14:25:46.544 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3587700127}) 2024-09-18T14:25:46.544 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.544 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2605566472}) 2024-09-18T14:25:46.544 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.535+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2790614830}) 2024-09-18T14:25:46.546 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.539+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.546 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.539+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=2735869755}) 2024-09-18T14:25:46.546 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.539+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=64} crc {osdmap=2924959355}) 2024-09-18T14:25:46.546 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.539+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:25:46.546 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.539+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=2735869755}) 2024-09-18T14:25:46.546 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:25:46.539+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=63} crc {osdmap=3433173835}) 2024-09-18T14:25:47.540 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T14:25:47.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient: get_monmap_and_config 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient: build_initial_monmap 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient: monmap: 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:47.650404+0000 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:47.650404+0000 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:47.652 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 Processor -- start 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 -- start start 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient: init 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f80a406e780 0x7f80a406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f80a406e780 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a412e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f80a412e400 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:47.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80a409c230 0x7f80a409c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f80a409c230 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a409d6a0 con 0x7f80a409c230 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a406cf70 con 0x7f80a406e780 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a406cc90 con 0x7f80a412e400 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.643+0000 7f80a3fff700 10 monclient(hunting): authenticate will time out at 1068.639982s 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80a409c230 0x7f80a409c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a412e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80a409c230 0x7f80a409c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39314/0 (socket says 172.21.15.104:39314) 2024-09-18T14:25:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 -- 172.21.15.104:0/288463696 learned_addr learned my addr 172.21.15.104:0/288463696 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): get_auth_request con 0x7f80a409c230 auth_method 0 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 10 monclient(hunting): get_auth_request con 0x7f80a412e400 auth_method 0 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a2ffd700 1 --2- 172.21.15.104:0/288463696 >> v2:172.21.15.174:3300/0 conn(0x7f80a406e780 0x7f80a406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a2ffd700 10 monclient(hunting): get_auth_request con 0x7f80a406e780 auth_method 0 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient(hunting): handle_auth_done global_id 14661 payload 274 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 -- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 msgr2=0x7f80a412e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 --2- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a412e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 -- 172.21.15.104:0/288463696 >> v2:172.21.15.174:3300/0 conn(0x7f80a406e780 msgr2=0x7f80a406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 --2- 172.21.15.104:0/288463696 >> v2:172.21.15.174:3300/0 conn(0x7f80a406e780 0x7f80a406eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient: _finish_hunting 0 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 monclient: found mon.noname-a 2024-09-18T14:25:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 -- 172.21.15.104:0/288463696 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a406d250 con 0x7f80a409c230 2024-09-18T14:25:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a2ffd700 1 --2- 172.21.15.104:0/288463696 >> v2:172.21.15.174:3300/0 conn(0x7f80a406e780 0x7f80a406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:25:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 -- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a409c230 msgr2=0x7f80a409c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a37fe700 1 --2- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a409c230 0x7f80a409c650 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f80a4065630 tx=0x7f809c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26bd7a74289d120 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:47.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 1 -- 172.21.15.104:0/288463696 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c009e70 con 0x7f80a409c230 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: dump: 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:47.664 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: _finish_auth 0 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 1 -- 172.21.15.104:0/288463696 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c00ba70 con 0x7f80a409c230 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a3fff700 5 monclient: authenticate success, global_id 14661 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a3fff700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a3fff700 4 set_mon_vals no callback set 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 1 -- 172.21.15.104:0/288463696 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c00be30 con 0x7f80a409c230 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:47.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.647+0000 7f80a1ffb700 10 monclient: dump: 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient: shutdown 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a409c230 msgr2=0x7f80a409c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:47.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 --2- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a409c230 0x7f80a409c650 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f80a4065630 tx=0x7f809c009250 comp rx=0 tx=0).stop 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- 172.21.15.104:0/288463696 shutdown_connections 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 --2- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a409c230 0x7f80a409c650 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 --2- 172.21.15.104:0/288463696 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a412e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 --2- 172.21.15.104:0/288463696 >> v2:172.21.15.174:3300/0 conn(0x7f80a406e780 0x7f80a406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- 172.21.15.104:0/288463696 >> 172.21.15.104:0/288463696 conn(0x7f80a4068160 msgr2=0x7f80a40a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- 172.21.15.104:0/288463696 shutdown_connections 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- 172.21.15.104:0/288463696 wait complete. 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient: build_initial_monmap 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient: monmap: 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:47.667 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 Processor -- start 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- start start 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient: init 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a406e780 0x7f80a4141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f80a406e780 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a409c230 0x7f80a4141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f80a409c230 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a4141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f80a412e400 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a409be90 con 0x7f80a406e780 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a40a14f0 con 0x7f80a409c230 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a411f980 con 0x7f80a412e400 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a3fff700 10 monclient(hunting): authenticate will time out at 1068.646397s 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a4141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a406e780 0x7f80a4141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a4141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46686/0 (socket says 172.21.15.104:46686) 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 -- 172.21.15.104:0/337589197 learned_addr learned my addr 172.21.15.104:0/337589197 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): get_auth_request con 0x7f80a412e400 auth_method 0 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 10 monclient(hunting): get_auth_request con 0x7f80a406e780 auth_method 0 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:47.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a27fc700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a409c230 0x7f80a4141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a27fc700 10 monclient(hunting): get_auth_request con 0x7f80a409c230 auth_method 0 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient(hunting): handle_auth_done global_id 24119 payload 824 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 -- 172.21.15.104:0/337589197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a409c230 msgr2=0x7f80a4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a409c230 0x7f80a4141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 -- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a406e780 msgr2=0x7f80a4141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a406e780 0x7f80a4141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:47.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient: _finish_hunting 0 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 monclient: found mon.noname-c 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.651+0000 7f80a37fe700 1 -- 172.21.15.104:0/337589197 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c0094f0 con 0x7f80a412e400 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a37fe700 1 -- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 msgr2=0x7f80a4141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a37fe700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a4141d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f809c0090b0 tx=0x7f809c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 1 -- 172.21.15.104:0/337589197 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c019e40 con 0x7f80a412e400 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: dump: 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: _finish_auth 0 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 5 monclient: authenticate success, global_id 24119 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 1 -- 172.21.15.104:0/337589197 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c006a90 con 0x7f80a412e400 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 10 monclient: _renew_subs 2024-09-18T14:25:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 1 -- 172.21.15.104:0/337589197 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c0108d0 con 0x7f80a412e400 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a411f470 con 0x7f80a412e400 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a17fa700 4 set_mon_vals no callback set 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 10 monclient: dump: 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 10 monclient: _renew_subs 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a41221c0 con 0x7f80a412e400 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a41219f0 con 0x7f80a412e400 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f80a37fe700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a4141d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f809c0090b0 tx=0x7f809c019000 comp rx=0 tx=0).queue_received will delay after 1.72667e+09 on 0x7f809c014070 osd_map(232..232 src has 1..232) v4 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 1 -- 172.21.15.104:0/337589197 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f809c0183e0 con 0x7f80a412e400 2024-09-18T14:25:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.655+0000 7f808affd700 1 --2- 172.21.15.104:0/337589197 >> v2:172.21.15.174:6812/17603 conn(0x7f8090035a80 0x7f8090037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:47.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.659+0000 7f80a2ffd700 1 --2- 172.21.15.104:0/337589197 >> v2:172.21.15.174:6812/17603 conn(0x7f8090035a80 0x7f8090037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:47.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.659+0000 7f80a2ffd700 10 monclient: get_auth_request con 0x7f8090035a80 auth_method 0 2024-09-18T14:25:47.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.659+0000 7f80a2ffd700 1 --2- 172.21.15.104:0/337589197 >> v2:172.21.15.174:6812/17603 conn(0x7f8090035a80 0x7f8090037f30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8098006f00 tx=0x7f8098008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:47.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:47.667+0000 7f808affd700 1 -- 172.21.15.104:0/337589197 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f809c014070 con 0x7f80a412e400 2024-09-18T14:25:48.148 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:48.148 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21249]: starting. 2024-09-18T14:25:48.149 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21249]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21214 2024-09-18T14:25:48.149 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21249]: created object 0... 2024-09-18T14:25:48.149 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21249]: created object 25... 2024-09-18T14:25:48.149 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21249]: created object 49... 2024-09-18T14:25:48.149 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21249]: finishing. 2024-09-18T14:25:48.149 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21249]: shutting down. 2024-09-18T14:25:48.212 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:48.213 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21251]: starting. 2024-09-18T14:25:48.213 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21251]: listing objects. 2024-09-18T14:25:48.213 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21251]: listed object 0... 2024-09-18T14:25:48.213 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21251]: listed object 25... 2024-09-18T14:25:48.213 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21251]: saw 50 objects 2024-09-18T14:25:48.213 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21251]: shutting down. 2024-09-18T14:25:48.489 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.483+0000 7f808affd700 1 -- 172.21.15.104:0/337589197 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f809c022b30 con 0x7f80a412e400 2024-09-18T14:25:48.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.483+0000 7f808affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:48.490 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.483+0000 7f808affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:48.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.711+0000 7f80a3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:48.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.711+0000 7f80a3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:48.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.711+0000 7f80a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:25:48.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.711+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f80a40633b0 con 0x7f80a412e400 2024-09-18T14:25:48.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.711+0000 7f808affd700 1 -- 172.21.15.104:0/337589197 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f809c022350 con 0x7f80a412e400 2024-09-18T14:25:48.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.711+0000 7f808affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:48.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.711+0000 7f808affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:48.718 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 >> v2:172.21.15.174:6812/17603 conn(0x7f8090035a80 msgr2=0x7f8090037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 --2- 172.21.15.104:0/337589197 >> v2:172.21.15.174:6812/17603 conn(0x7f8090035a80 0x7f8090037f30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8098006f00 tx=0x7f8098008040 comp rx=0 tx=0).stop 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 10 monclient: shutdown 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 msgr2=0x7f80a4141d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a4141d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f809c0090b0 tx=0x7f809c019000 comp rx=0 tx=0).stop 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 shutdown_connections 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 --2- 172.21.15.104:0/337589197 >> v2:172.21.15.174:6812/17603 conn(0x7f8090035a80 0x7f8090037f30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f80a412e400 0x7f80a4141d90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80a409c230 0x7f80a4141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 --2- 172.21.15.104:0/337589197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80a406e780 0x7f80a4141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.721 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 >> 172.21.15.104:0/337589197 conn(0x7f80a4068160 msgr2=0x7f80a40a1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:48.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 shutdown_connections 2024-09-18T14:25:48.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.715+0000 7f80a3fff700 1 -- 172.21.15.104:0/337589197 wait complete. 2024-09-18T14:25:48.732 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T14:25:48.732 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T14:25:48.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T14:25:48.732 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:25:48.771 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:48.771 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21250]: starting. 2024-09-18T14:25:48.771 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21250]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21214 2024-09-18T14:25:48.771 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21250]: shutting down. 2024-09-18T14:25:48.772 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:48.772 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:25:48.773 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:25:48.775 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:25:48.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient: get_monmap_and_config 2024-09-18T14:25:48.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient: build_initial_monmap 2024-09-18T14:25:48.841 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient: monmap: 2024-09-18T14:25:48.841 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:48.841 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:48.841 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:48.840725+0000 2024-09-18T14:25:48.841 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:48.840725+0000 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:48.842 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 1 Processor -- start 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 1 -- start start 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient: init 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbab809e830 0x7fbab8131d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient(hunting): picked mon.noname-a con 0x7fbab809e830 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbab8132260 0x7fbab809ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient(hunting): picked mon.noname-c con 0x7fbab8132260 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbab809f140 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab809b7a0 con 0x7fbab809f140 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab8099f10 con 0x7fbab809e830 2024-09-18T14:25:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab809b4c0 con 0x7fbab8132260 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabe335700 10 monclient(hunting): authenticate will time out at 1069.831208s 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbab809e830 0x7fbab8131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbab809e830 0x7fbab8131d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39332/0 (socket says 172.21.15.104:39332) 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 1 -- 172.21.15.104:0/1421462760 learned_addr learned my addr 172.21.15.104:0/1421462760 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 10 monclient(hunting): get_auth_request con 0x7fbab809e830 auth_method 0 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 1 --2- 172.21.15.104:0/1421462760 >> v2:172.21.15.174:3300/0 conn(0x7fbab8132260 0x7fbab809ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 10 monclient(hunting): get_auth_request con 0x7fbab8132260 auth_method 0 2024-09-18T14:25:48.846 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabd333700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.835+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 10 monclient(hunting): handle_auth_done global_id 14682 payload 274 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 1 -- 172.21.15.104:0/1421462760 >> v2:172.21.15.174:3300/0 conn(0x7fbab8132260 msgr2=0x7fbab809ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 1 --2- 172.21.15.104:0/1421462760 >> v2:172.21.15.174:3300/0 conn(0x7fbab8132260 0x7fbab809ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 1 -- 172.21.15.104:0/1421462760 >> v1:172.21.15.174:6789/0 conn(0x7fbab809f140 legacy=0x7fbab809fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 10 monclient: _finish_hunting 0 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 1 monclient: found mon.noname-a 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 1 -- 172.21.15.104:0/1421462760 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab809ba80 con 0x7fbab809e830 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabcb32700 1 --2- 172.21.15.104:0/1421462760 >> v2:172.21.15.174:3300/0 conn(0x7fbab8132260 0x7fbab809ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 1 -- 172.21.15.104:0/1421462760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab809e830 msgr2=0x7fbab8131d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:48.848 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.839+0000 7fbabd333700 1 --2- 172.21.15.104:0/1421462760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab809e830 0x7fbab8131d20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbab809c4d0 tx=0x7fbab4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4923a78363080822 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 1 -- 172.21.15.104:0/1421462760 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4009e70 con 0x7fbab809e830 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: dump: 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:48.864 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: _finish_auth 0 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbabe335700 5 monclient: authenticate success, global_id 14682 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbabe335700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 1 -- 172.21.15.104:0/1421462760 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab400ba30 con 0x7fbab809e830 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 1 -- 172.21.15.104:0/1421462760 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab400bdf0 con 0x7fbab809e830 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbabe335700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbabe335700 4 set_mon_vals no callback set 2024-09-18T14:25:48.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.855+0000 7fbaaffff700 10 monclient: dump: 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:48.866 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 10 monclient: shutdown 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 -- 172.21.15.104:0/1421462760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab809e830 msgr2=0x7fbab8131d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 --2- 172.21.15.104:0/1421462760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab809e830 0x7fbab8131d20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbab809c4d0 tx=0x7fbab4009250 comp rx=0 tx=0).stop 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 -- 172.21.15.104:0/1421462760 shutdown_connections 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 --2- 172.21.15.104:0/1421462760 >> v2:172.21.15.174:3300/0 conn(0x7fbab8132260 0x7fbab809ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 --2- 172.21.15.104:0/1421462760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab809e830 0x7fbab8131d20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 -- 172.21.15.104:0/1421462760 >> 172.21.15.104:0/1421462760 conn(0x7fbab80676a0 msgr2=0x7fbab8067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 -- 172.21.15.104:0/1421462760 shutdown_connections 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 -- 172.21.15.104:0/1421462760 wait complete. 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 10 monclient: build_initial_monmap 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 10 monclient: monmap: 2024-09-18T14:25:48.868 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 Processor -- start 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 1 -- start start 2024-09-18T14:25:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.859+0000 7fbabe335700 10 monclient: init 2024-09-18T14:25:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 0x7fbab81314e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): picked mon.noname-c con 0x7fbab809e830 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbab809f140 0x7fbab806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): picked mon.noname-b con 0x7fbab809f140 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8132260 0x7fbab806c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): picked mon.noname-a con 0x7fbab8132260 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab809f5e0 con 0x7fbab8132260 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab806dc40 con 0x7fbab809f140 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab806d7d0 con 0x7fbab809e830 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabe335700 10 monclient(hunting): authenticate will time out at 1069.857240s 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 0x7fbab81314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8132260 0x7fbab806c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 0x7fbab81314e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46702/0 (socket says 172.21.15.104:46702) 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 -- 172.21.15.104:0/3010353205 learned_addr learned my addr 172.21.15.104:0/3010353205 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): get_auth_request con 0x7fbab809e830 auth_method 0 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8132260 0x7fbab806c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39340/0 (socket says 172.21.15.104:39340) 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 10 monclient(hunting): get_auth_request con 0x7fbab8132260 auth_method 0 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbab809f140 0x7fbab806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 10 monclient(hunting): get_auth_request con 0x7fbab809f140 auth_method 0 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabdb34700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:48.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient(hunting): handle_auth_done global_id 24128 payload 824 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbab809f140 msgr2=0x7fbab806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbab809f140 0x7fbab806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8132260 msgr2=0x7fbab806c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8132260 0x7fbab806c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient: _finish_hunting 0 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 monclient: found mon.noname-c 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:25:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.863+0000 7fbabd333700 1 -- 172.21.15.104:0/3010353205 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa8009420 con 0x7fbab809e830 2024-09-18T14:25:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.867+0000 7fbabd333700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 msgr2=0x7fbab81314e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.867+0000 7fbabd333700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 0x7fbab81314e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbab400b3c0 tx=0x7fbab4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab401f070 con 0x7fbab809e830 2024-09-18T14:25:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:25:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: dump: 2024-09-18T14:25:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: _finish_auth 0 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 5 monclient: authenticate success, global_id 24128 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4011940 con 0x7fbab809e830 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 10 monclient: _renew_subs 2024-09-18T14:25:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab40094f0 con 0x7fbab809e830 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4011ce0 con 0x7fbab809e830 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaad7fa700 10 monclient: dump: 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:48.880 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbaaeffd700 4 set_mon_vals no callback set 2024-09-18T14:25:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 10 monclient: _renew_subs 2024-09-18T14:25:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:25:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab806e300 con 0x7fbab809e830 2024-09-18T14:25:48.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 0x7fbab806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab806f390 con 0x7fbab8124130 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 0x7fbab806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient: get_auth_request con 0x7fbab8124130 auth_method 0 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 10 monclient(hunting): handle_auth_done global_id 14688 payload 824 2024-09-18T14:25:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.871+0000 7fbabcb32700 1 -- 172.21.15.104:0/3010353205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbab806e4f0 con 0x7fbab8124130 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fbab4023070 con 0x7fbab809e830 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 --2- 172.21.15.104:0/3010353205 >> v2:172.21.15.174:6812/17603 conn(0x7fba9c035ad0 0x7fba9c037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbab4049020 con 0x7fbab809e830 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbabcb32700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 msgr2=0x7fbab806c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbabcb32700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 0x7fbab806c7d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8011c20 tx=0x7fbaa80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbabdb34700 1 --2- 172.21.15.104:0/3010353205 >> v2:172.21.15.174:6812/17603 conn(0x7fba9c035ad0 0x7fba9c037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbabdb34700 10 monclient: get_auth_request con 0x7fba9c035ad0 auth_method 0 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8014440 con 0x7fbab8124130 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 10 monclient: dump: 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbaa80145a0 con 0x7fbab8124130 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 msgr2=0x7fbab806c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 0x7fbab806c7d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8011c20 tx=0x7fbaa80091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 msgr2=0x7fbab806c7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.875+0000 7fbaad7fa700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8124130 0x7fbab806c7d0 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8011c20 tx=0x7fbaa80091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.879+0000 7fbabdb34700 1 --2- 172.21.15.104:0/3010353205 >> v2:172.21.15.174:6812/17603 conn(0x7fba9c035ad0 0x7fba9c037f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbab0007fd0 tx=0x7fbab0002620 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabe335700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabe335700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 0x7fbab80883f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabe335700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab8064840 con 0x7fbab8087ff0 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 0x7fbab80883f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient: get_auth_request con 0x7fbab8087ff0 auth_method 0 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 10 monclient(hunting): handle_auth_done global_id 14691 payload 824 2024-09-18T14:25:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.883+0000 7fbabcb32700 1 -- 172.21.15.104:0/3010353205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbab8063450 con 0x7fbab8087ff0 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabcb32700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 msgr2=0x7fbab80883f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabcb32700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 0x7fbab80883f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8011c20 tx=0x7fbaa801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8028400 con 0x7fbab8087ff0 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 10 monclient: dump: 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.894 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:48.896 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:48.896 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:48.896 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:48.896 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:48.896 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:48.896 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fbaa801c3e0 con 0x7fbab8087ff0 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 msgr2=0x7fbab80883f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 0x7fbab80883f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8011c20 tx=0x7fbaa801c000 comp rx=0 tx=0).stop 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 msgr2=0x7fbab80883f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbaad7fa700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbab8087ff0 0x7fbab80883f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.897 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 26, 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 7, 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:25:48.898 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:48.899 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:25:48.900 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:48.903 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:48.904 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:25:48.905 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:25:48.906 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:48.907 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:25:48.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:48.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:48.910 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:25:48.910 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:25:48.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 >> v2:172.21.15.174:6812/17603 conn(0x7fba9c035ad0 msgr2=0x7fba9c037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 --2- 172.21.15.104:0/3010353205 >> v2:172.21.15.174:6812/17603 conn(0x7fba9c035ad0 0x7fba9c037f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbab0007fd0 tx=0x7fbab0002620 comp rx=0 tx=0).stop 2024-09-18T14:25:48.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 10 monclient: shutdown 2024-09-18T14:25:48.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 msgr2=0x7fbab81314e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 0x7fbab81314e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbab400b3c0 tx=0x7fbab4017000 comp rx=0 tx=0).stop 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabdb34700 1 -- 172.21.15.104:0/3010353205 reap_dead start 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 shutdown_connections 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 --2- 172.21.15.104:0/3010353205 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbab809e830 0x7fbab81314e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 >> 172.21.15.104:0/3010353205 conn(0x7fbab80676a0 msgr2=0x7fbab809ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 shutdown_connections 2024-09-18T14:25:48.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:48.887+0000 7fbabe335700 1 -- 172.21.15.104:0/3010353205 wait complete. 2024-09-18T14:25:48.912 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:25:48.912 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T14:25:48.913 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient: get_monmap_and_config 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient: build_initial_monmap 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient: monmap: 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:49.020506+0000 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:49.020506+0000 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:49.021 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:49.022 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:49.022 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:49.022 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:49.022 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 1 Processor -- start 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 1 -- start start 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient: init 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4e0097c10 addr v1:172.21.15.104:6789/0 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb4e0096300 0x7fb4e009c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient(hunting): picked mon.noname-c con 0x7fb4e0096300 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4e0135390 0x7fb4e0137770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient(hunting): picked mon.noname-a con 0x7fb4e0135390 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e0097fe0 con 0x7fb4e0097c10 2024-09-18T14:25:49.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e0096700 con 0x7fb4e0135390 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e009cf10 con 0x7fb4e0096300 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e57bc700 10 monclient(hunting): authenticate will time out at 1070.010141s 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4e0135390 0x7fb4e0137770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4e0135390 0x7fb4e0137770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39380/0 (socket says 172.21.15.104:39380) 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 1 -- 172.21.15.104:0/2659497623 learned_addr learned my addr 172.21.15.104:0/2659497623 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 10 monclient(hunting): get_auth_request con 0x7fb4e0135390 auth_method 0 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 1 --2- 172.21.15.104:0/2659497623 >> v2:172.21.15.174:3300/0 conn(0x7fb4e0096300 0x7fb4e009c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4e4fbb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): get_auth_request con 0x7fb4e0096300 auth_method 0 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_done global_id 5383 payload 274 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 1 -- 172.21.15.104:0/2659497623 >> v2:172.21.15.104:3300/0 conn(0x7fb4e0135390 msgr2=0x7fb4e0137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 1 --2- 172.21.15.104:0/2659497623 >> v2:172.21.15.104:3300/0 conn(0x7fb4e0135390 0x7fb4e0137770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 1 -- 172.21.15.104:0/2659497623 >> v1:172.21.15.104:6789/0 conn(0x7fb4e0097c10 legacy=0x7fb4e009bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:49.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:25:49.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 1 monclient: found mon.noname-c 2024-09-18T14:25:49.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:49.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:25:49.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.015+0000 7fb4df7fe700 1 -- 172.21.15.104:0/2659497623 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e0095640 con 0x7fb4e0096300 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4df7fe700 1 -- 172.21.15.104:0/2659497623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0096300 msgr2=0x7fb4e009c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4df7fe700 1 --2- 172.21.15.104:0/2659497623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0096300 0x7fb4e009c2f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb4e009dcb0 tx=0x7fb4d0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ccc204767869c6b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 1 -- 172.21.15.104:0/2659497623 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d000e070 con 0x7fb4e0096300 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: dump: 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: _finish_auth 0 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 1 -- 172.21.15.104:0/2659497623 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d000bd70 con 0x7fb4e0096300 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 5 monclient: authenticate success, global_id 5383 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 1 -- 172.21.15.104:0/2659497623 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0012680 con 0x7fb4e0096300 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 4 set_mon_vals no callback set 2024-09-18T14:25:49.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4deffd700 10 monclient: dump: 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 10 monclient: shutdown 2024-09-18T14:25:49.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 -- 172.21.15.104:0/2659497623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0096300 msgr2=0x7fb4e009c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/2659497623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0096300 0x7fb4e009c2f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb4e009dcb0 tx=0x7fb4d0009780 comp rx=0 tx=0).stop 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 -- 172.21.15.104:0/2659497623 shutdown_connections 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/2659497623 >> v2:172.21.15.104:3300/0 conn(0x7fb4e0135390 0x7fb4e0137770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/2659497623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0096300 0x7fb4e009c2f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 -- 172.21.15.104:0/2659497623 >> 172.21.15.104:0/2659497623 conn(0x7fb4e00a11f0 msgr2=0x7fb4e00a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 -- 172.21.15.104:0/2659497623 shutdown_connections 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 1 -- 172.21.15.104:0/2659497623 wait complete. 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 10 monclient: build_initial_monmap 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.019+0000 7fb4e57bc700 10 monclient: monmap: 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:49.036 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 Processor -- start 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 -- start start 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient: init 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4e0096300 0x7fb4e006d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.037 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): picked mon.noname-c con 0x7fb4e0096300 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 0x7fb4e00986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): picked mon.noname-a con 0x7fb4e0097c10 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0135390 0x7fb4e0098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): picked mon.noname-b con 0x7fb4e0135390 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e012ff90 con 0x7fb4e0097c10 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e012b3b0 con 0x7fb4e0135390 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e012b0d0 con 0x7fb4e0096300 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e57bc700 10 monclient(hunting): authenticate will time out at 1070.016652s 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 0x7fb4e00986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.038 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 0x7fb4e00986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39388/0 (socket says 172.21.15.104:39388) 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 -- 172.21.15.104:0/3101838791 learned_addr learned my addr 172.21.15.104:0/3101838791 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): get_auth_request con 0x7fb4e0097c10 auth_method 0 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4e0096300 0x7fb4e006d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 10 monclient(hunting): get_auth_request con 0x7fb4e0096300 auth_method 0 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0135390 0x7fb4e0098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 10 monclient(hunting): get_auth_request con 0x7fb4e0135390 auth_method 0 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4e4fbb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient(hunting): handle_auth_done global_id 14697 payload 824 2024-09-18T14:25:49.040 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4e0096300 msgr2=0x7fb4e006d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4e0096300 0x7fb4e006d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0135390 msgr2=0x7fb4e0098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4e0135390 0x7fb4e0098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 monclient: found mon.noname-a 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 -- 172.21.15.104:0/3101838791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d0009a20 con 0x7fb4e0097c10 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 msgr2=0x7fb4e00986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.023+0000 7fb4df7fe700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 0x7fb4e00986d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00097c0 tx=0x7fb4d0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0012e50 con 0x7fb4e0097c10 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.041 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: dump: 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: _finish_auth 0 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:49.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 5 monclient: authenticate success, global_id 14697 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 10 monclient: _renew_subs 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e012b690 con 0x7fb4e0097c10 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 10 monclient: _renew_subs 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e009a520 con 0x7fb4e0097c10 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d0027470 con 0x7fb4e0097c10 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0027830 con 0x7fb4e0097c10 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 10 monclient: dump: 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.043 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4ddffb700 4 set_mon_vals no callback set 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 0x7fb4c4004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c4007cd0 con 0x7fb4c4002860 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 0x7fb4c4004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fb4d0024070 con 0x7fb4e0097c10 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient: get_auth_request con 0x7fb4c4002860 auth_method 0 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 1 --2- 172.21.15.104:0/3101838791 >> v2:172.21.15.174:6812/17603 conn(0x7fb4c0035ab0 0x7fb4c0037f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb4d004a370 con 0x7fb4e0097c10 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 10 monclient(hunting): handle_auth_done global_id 24134 payload 824 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4dffff700 1 -- 172.21.15.104:0/3101838791 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4c4007e90 con 0x7fb4c4002860 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e4fbb700 1 --2- 172.21.15.104:0/3101838791 >> v2:172.21.15.174:6812/17603 conn(0x7fb4c0035ab0 0x7fb4c0037f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.027+0000 7fb4e4fbb700 10 monclient: get_auth_request con 0x7fb4c0035ab0 auth_method 0 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4dffff700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 msgr2=0x7fb4c4004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4dffff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 0x7fb4c4004cf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800f7b0 tx=0x7fb4c8007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8014890 con 0x7fb4c4002860 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 10 monclient: dump: 2024-09-18T14:25:49.048 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4e4fbb700 1 --2- 172.21.15.104:0/3101838791 >> v2:172.21.15.174:6812/17603 conn(0x7fb4c0035ab0 0x7fb4c0037f60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8006d40 tx=0x7fb4d8009250 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb4c80149f0 con 0x7fb4c4002860 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:49.049 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 msgr2=0x7fb4c4004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 0x7fb4c4004cf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800f7b0 tx=0x7fb4c8007ea0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 msgr2=0x7fb4c4004cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:49.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.031+0000 7fb4d7fff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4002860 0x7fb4c4004cf0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800f7b0 tx=0x7fb4c8007ea0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4e57bc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 0x7fb4c4007140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4e57bc700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c4007780 con 0x7fb4c4006cc0 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 0x7fb4c4007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient: get_auth_request con 0x7fb4c4006cc0 auth_method 0 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.051 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 10 monclient(hunting): handle_auth_done global_id 24137 payload 824 2024-09-18T14:25:49.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 1 -- 172.21.15.104:0/3101838791 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb4c4007940 con 0x7fb4c4006cc0 2024-09-18T14:25:49.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 msgr2=0x7fb4c4007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4dffff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 0x7fb4c4007140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800f7b0 tx=0x7fb4c8019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c801fa10 con 0x7fb4c4006cc0 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4d7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.043+0000 7fb4d7fff700 10 monclient: dump: 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fb4c8024070 con 0x7fb4c4006cc0 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4d7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4d7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 msgr2=0x7fb4c4007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4d7fff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 0x7fb4c4007140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800f7b0 tx=0x7fb4c8019000 comp rx=0 tx=0).stop 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4d7fff700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 msgr2=0x7fb4c4007140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:49.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4d7fff700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb4c4006cc0 0x7fb4c4007140 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 26, 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 8, 2024-09-18T14:25:49.055 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:25:49.056 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:25:49.057 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:49.058 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.059 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:25:49.060 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:49.061 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:25:49.062 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:49.065 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.066 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.066 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:25:49.066 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:25:49.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 >> v2:172.21.15.174:6812/17603 conn(0x7fb4c0035ab0 msgr2=0x7fb4c0037f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/3101838791 >> v2:172.21.15.174:6812/17603 conn(0x7fb4c0035ab0 0x7fb4c0037f60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8006d40 tx=0x7fb4d8009250 comp rx=0 tx=0).stop 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 10 monclient: shutdown 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 msgr2=0x7fb4e00986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e4fbb700 1 -- 172.21.15.104:0/3101838791 reap_dead start 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 0x7fb4e00986d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00097c0 tx=0x7fb4d0018000 comp rx=0 tx=0).stop 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 shutdown_connections 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 --2- 172.21.15.104:0/3101838791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4e0097c10 0x7fb4e00986d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 >> 172.21.15.104:0/3101838791 conn(0x7fb4e00a11f0 msgr2=0x7fb4e009dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 shutdown_connections 2024-09-18T14:25:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.047+0000 7fb4e57bc700 1 -- 172.21.15.104:0/3101838791 wait complete. 2024-09-18T14:25:49.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:25:49.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T14:25:49.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:25:49.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient: get_monmap_and_config 2024-09-18T14:25:49.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient: build_initial_monmap 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient: monmap: 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:49.176537+0000 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:49.176537+0000 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:49.178 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:49.179 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 1 Processor -- start 2024-09-18T14:25:49.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 1 -- start start 2024-09-18T14:25:49.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient: init 2024-09-18T14:25:49.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:49.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T14:25:49.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient(hunting): picked mon.noname-b con 0x7ff5f006b050 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5f006b420 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff5f0136f10 0x7ff5f01392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient(hunting): picked mon.noname-c con 0x7ff5f0136f10 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f009f7b0 con 0x7ff5f006b420 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f009deb0 con 0x7ff5f0136f10 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f009e0f0 con 0x7ff5f006b050 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5effff700 10 monclient(hunting): authenticate will time out at 1070.166091s 2024-09-18T14:25:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f006a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46762/0 (socket says 172.21.15.104:46762) 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 1 -- 172.21.15.104:0/4271257747 learned_addr learned my addr 172.21.15.104:0/4271257747 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 10 monclient(hunting): get_auth_request con 0x7ff5f006b050 auth_method 0 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 1 --2- 172.21.15.104:0/4271257747 >> v2:172.21.15.174:3300/0 conn(0x7ff5f0136f10 0x7ff5f01392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): get_auth_request con 0x7ff5f0136f10 auth_method 0 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient(hunting): handle_auth_done global_id 5389 payload 274 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 1 -- 172.21.15.104:0/4271257747 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 msgr2=0x7ff5f006a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 1 --2- 172.21.15.104:0/4271257747 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f006a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 1 -- 172.21.15.104:0/4271257747 >> v1:172.21.15.174:6789/0 conn(0x7ff5f006b420 legacy=0x7ff5f006b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:49.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:25:49.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 1 monclient: found mon.noname-c 2024-09-18T14:25:49.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:49.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:25:49.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5ef7fe700 1 -- 172.21.15.104:0/4271257747 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f009f4d0 con 0x7ff5f0136f10 2024-09-18T14:25:49.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.171+0000 7ff5eeffd700 1 --2- 172.21.15.104:0/4271257747 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f006a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:25:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5ef7fe700 1 -- 172.21.15.104:0/4271257747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 msgr2=0x7ff5f01392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5ef7fe700 1 --2- 172.21.15.104:0/4271257747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 0x7ff5f01392f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00a11c0 tx=0x7ff5e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4168154faaff212b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 1 -- 172.21.15.104:0/4271257747 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8011840 con 0x7ff5f0136f10 2024-09-18T14:25:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:25:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: dump: 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.194 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: _finish_auth 0 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 5 monclient: authenticate success, global_id 5389 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 1 -- 172.21.15.104:0/4271257747 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e80119a0 con 0x7ff5f0136f10 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 4 set_mon_vals no callback set 2024-09-18T14:25:49.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 1 -- 172.21.15.104:0/4271257747 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8011d40 con 0x7ff5f0136f10 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5edffb700 10 monclient: dump: 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.196 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 10 monclient: shutdown 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 -- 172.21.15.104:0/4271257747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 msgr2=0x7ff5f01392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 --2- 172.21.15.104:0/4271257747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 0x7ff5f01392f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00a11c0 tx=0x7ff5e80091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 -- 172.21.15.104:0/4271257747 shutdown_connections 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 --2- 172.21.15.104:0/4271257747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 0x7ff5f01392f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 --2- 172.21.15.104:0/4271257747 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f006a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 -- 172.21.15.104:0/4271257747 >> 172.21.15.104:0/4271257747 conn(0x7ff5f00a3b80 msgr2=0x7ff5f009fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:49.197 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 -- 172.21.15.104:0/4271257747 shutdown_connections 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.175+0000 7ff5effff700 1 -- 172.21.15.104:0/4271257747 wait complete. 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient: build_initial_monmap 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient: monmap: 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:49.198 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 Processor -- start 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 -- start start 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient: init 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f0135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): picked mon.noname-c con 0x7ff5f006b050 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5f006b420 0x7ff5f011e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): picked mon.noname-a con 0x7ff5f006b420 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:49.199 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 0x7ff5f011edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): picked mon.noname-b con 0x7ff5f0136f10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f006f3e0 con 0x7ff5f006b420 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f00a1df0 con 0x7ff5f0136f10 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0137830 con 0x7ff5f006b050 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5effff700 10 monclient(hunting): authenticate will time out at 1070.173252s 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f0135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f0135d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:46774/0 (socket says 172.21.15.104:46774) 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 1 -- 172.21.15.104:0/1644729569 learned_addr learned my addr 172.21.15.104:0/1644729569 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 10 monclient(hunting): get_auth_request con 0x7ff5f006b050 auth_method 0 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5f006b420 0x7ff5f011e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): get_auth_request con 0x7ff5f006b420 auth_method 0 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 0x7ff5f011edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 10 monclient(hunting): get_auth_request con 0x7ff5f0136f10 auth_method 0 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient(hunting): handle_auth_done global_id 14700 payload 824 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 msgr2=0x7ff5f0135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f0135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 msgr2=0x7ff5f011edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136f10 0x7ff5f011edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient: _finish_hunting 0 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 1 monclient: found mon.noname-a 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5ee7fc700 1 -- 172.21.15.104:0/1644729569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e40094f0 con 0x7ff5f006b420 2024-09-18T14:25:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.179+0000 7ff5eeffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7ff5f006b050 0x7ff5f0135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5ee7fc700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5f006b420 msgr2=0x7ff5f011e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5ee7fc700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5f006b420 0x7ff5f011e890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000c890 tx=0x7ff5e00026a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0012840 con 0x7ff5f006b420 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: dump: 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: _finish_auth 0 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 5 monclient: authenticate success, global_id 14700 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 10 monclient: _renew_subs 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e00129c0 con 0x7ff5f006b420 2024-09-18T14:25:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e8009420 con 0x7ff5f006b420 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0012d80 con 0x7ff5f006b420 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 10 monclient: dump: 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 10 monclient: _renew_subs 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5ed7fa700 4 set_mon_vals no callback set 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f0121e50 con 0x7ff5f006b420 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 0x7ff5f011f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0064960 con 0x7ff5f0141f20 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 0x7ff5f011f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient: get_auth_request con 0x7ff5f0141f20 auth_method 0 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7ff5e0011590 con 0x7ff5f006b420 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 1 --2- 172.21.15.104:0/1644729569 >> v2:172.21.15.174:6812/17603 conn(0x7ff5dc035a80 0x7ff5dc037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_done global_id 5395 payload 824 2024-09-18T14:25:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5eeffd700 1 -- 172.21.15.104:0/1644729569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5f0063450 con 0x7ff5f0141f20 2024-09-18T14:25:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff5e00481e0 con 0x7ff5f006b420 2024-09-18T14:25:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5ef7fe700 1 --2- 172.21.15.104:0/1644729569 >> v2:172.21.15.174:6812/17603 conn(0x7ff5dc035a80 0x7ff5dc037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.183+0000 7ff5ef7fe700 10 monclient: get_auth_request con 0x7ff5dc035a80 auth_method 0 2024-09-18T14:25:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5eeffd700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 msgr2=0x7ff5f011f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5eeffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 0x7ff5f011f310 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400f470 tx=0x7ff5e4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4014990 con 0x7ff5f0141f20 2024-09-18T14:25:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 10 monclient: dump: 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5ef7fe700 1 --2- 172.21.15.104:0/1644729569 >> v2:172.21.15.174:6812/17603 conn(0x7ff5dc035a80 0x7ff5dc037f30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80093f0 tx=0x7ff5e8017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff5e4014af0 con 0x7ff5f0141f20 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 msgr2=0x7ff5f011f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 0x7ff5f011f310 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400f470 tx=0x7ff5e4009250 comp rx=0 tx=0).stop 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 msgr2=0x7ff5f011f310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.187+0000 7ff5d6ffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0141f20 0x7ff5f011f310 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5effff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5effff700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 0x7ff5f011f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5effff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0096e50 con 0x7ff5f0136280 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 0x7ff5f011f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient: get_auth_request con 0x7ff5f0136280 auth_method 0 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 10 monclient(hunting): handle_auth_done global_id 5398 payload 824 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.199+0000 7ff5eeffd700 1 -- 172.21.15.104:0/1644729569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff5f0122370 con 0x7ff5f0136280 2024-09-18T14:25:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5eeffd700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 msgr2=0x7ff5f011f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5eeffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 0x7ff5f011f850 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400f470 tx=0x7ff5e401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4024040 con 0x7ff5f0136280 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 10 monclient: dump: 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.215 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7ff5e4014780 con 0x7ff5f0136280 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 msgr2=0x7ff5f011f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 0x7ff5f011f850 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400f470 tx=0x7ff5e401c000 comp rx=0 tx=0).stop 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 msgr2=0x7ff5f011f850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:25:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5d6ffd700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5f0136280 0x7ff5f011f850 secure :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400f470 tx=0x7ff5e401c000 comp rx=0 tx=0).stop 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 26, 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:25:49.217 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 8, 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:49.218 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:49.219 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:25:49.220 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:25:49.221 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:25:49.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:49.224 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.225 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.226 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:25:49.227 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:25:49.228 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:25:49.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 7 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:25:49.230 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:25:49.231 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:25:49.231 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:25:49.231 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:25:49.231 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:25:49.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:25:49.231 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:25:49.231 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 >> v2:172.21.15.174:6812/17603 conn(0x7ff5dc035a80 msgr2=0x7ff5dc037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 --2- 172.21.15.104:0/1644729569 >> v2:172.21.15.174:6812/17603 conn(0x7ff5dc035a80 0x7ff5dc037f30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80093f0 tx=0x7ff5e8017000 comp rx=0 tx=0).stop 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 10 monclient: shutdown 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5f006b420 msgr2=0x7ff5f011e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 --2- 172.21.15.104:0/1644729569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5f006b420 0x7ff5f011e890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000c890 tx=0x7ff5e00026a0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5ef7fe700 1 -- 172.21.15.104:0/1644729569 reap_dead start 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5ef7fe700 1 -- 172.21.15.104:0/1644729569 reap_dead start 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 shutdown_connections 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 >> 172.21.15.104:0/1644729569 conn(0x7ff5f00a3b80 msgr2=0x7ff5f006d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:49.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 shutdown_connections 2024-09-18T14:25:49.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.203+0000 7ff5effff700 1 -- 172.21.15.104:0/1644729569 wait complete. 2024-09-18T14:25:49.234 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:25:49.234 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T14:25:49.234 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T14:25:49.234 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T14:25:49.234 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T14:25:49.235 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T14:25:49.235 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:25:49.251 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T14:25:49.251 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T14:25:49.251 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:49.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient: get_monmap_and_config 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient: build_initial_monmap 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient: monmap: 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:49.401283+0000 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:49.401283+0000 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:49.402 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:49.403 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:49.403 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 Processor -- start 2024-09-18T14:25:49.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 -- start start 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient: init 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798095a50 0x7fe79806a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe798095a50 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe798095e20 0x7fe7980961f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): picked mon.noname-c con 0x7fe798095e20 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe798131cd0 0x7fe7981320f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe798131cd0 addr v2:172.21.15.104:3300/0 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79809fa20 con 0x7fe798131cd0 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79809f680 con 0x7fe798095e20 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe798069e60 con 0x7fe798095a50 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79759e700 10 monclient(hunting): authenticate will time out at 1070.391160s 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79659c700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798095a50 msgr2=0x7fe79806a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe79659c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798095a50 0x7fe79806a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe798131cd0 0x7fe7981320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe798131cd0 0x7fe7981320f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39410/0 (socket says 172.21.15.104:39410) 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 1 -- 172.21.15.104:0/3490732520 learned_addr learned my addr 172.21.15.104:0/3490732520 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:49.407 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): get_auth_request con 0x7fe798131cd0 auth_method 0 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 1 --2- 172.21.15.104:0/3490732520 >> v2:172.21.15.174:3300/0 conn(0x7fe798095e20 0x7fe7980961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 10 monclient(hunting): get_auth_request con 0x7fe798095e20 auth_method 0 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe78ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 10 monclient(hunting): handle_auth_done global_id 14703 payload 274 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 1 -- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798095a50 msgr2=0x7fe79806a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.395+0000 7fe796d9d700 1 --2- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798095a50 0x7fe79806a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 1 -- 172.21.15.104:0/3490732520 >> v2:172.21.15.174:3300/0 conn(0x7fe798095e20 msgr2=0x7fe7980961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 1 --2- 172.21.15.104:0/3490732520 >> v2:172.21.15.174:3300/0 conn(0x7fe798095e20 0x7fe7980961f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 1 monclient: found mon.noname-a 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 1 -- 172.21.15.104:0/3490732520 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79806a100 con 0x7fe798131cd0 2024-09-18T14:25:49.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe78ffff700 1 --2- 172.21.15.104:0/3490732520 >> v2:172.21.15.174:3300/0 conn(0x7fe798095e20 0x7fe7980961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 1 -- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798131cd0 msgr2=0x7fe7981320f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe796d9d700 1 --2- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798131cd0 0x7fe7981320f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe7980644f0 tx=0x7fe790009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f755de5e6fb5fe server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 1 -- 172.21.15.104:0/3490732520 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79000e070 con 0x7fe798131cd0 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: dump: 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.414 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: _finish_auth 0 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 1 -- 172.21.15.104:0/3490732520 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79000bc30 con 0x7fe798131cd0 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe79759e700 5 monclient: authenticate success, global_id 14703 2024-09-18T14:25:49.415 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe79759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe79759e700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 1 -- 172.21.15.104:0/3490732520 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7900092e0 con 0x7fe798131cd0 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe79759e700 4 set_mon_vals no callback set 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.399+0000 7fe795d9b700 10 monclient: dump: 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.416 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient: shutdown 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798131cd0 msgr2=0x7fe7981320f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 --2- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798131cd0 0x7fe7981320f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe7980644f0 tx=0x7fe790009840 comp rx=0 tx=0).stop 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- 172.21.15.104:0/3490732520 shutdown_connections 2024-09-18T14:25:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 --2- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798131cd0 0x7fe7981320f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 --2- 172.21.15.104:0/3490732520 >> v2:172.21.15.174:3300/0 conn(0x7fe798095e20 0x7fe7980961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 --2- 172.21.15.104:0/3490732520 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798095a50 0x7fe79806a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- 172.21.15.104:0/3490732520 >> 172.21.15.104:0/3490732520 conn(0x7fe7980a3120 msgr2=0x7fe7980a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- 172.21.15.104:0/3490732520 shutdown_connections 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- 172.21.15.104:0/3490732520 wait complete. 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient: build_initial_monmap 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient: monmap: 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:49.418 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 Processor -- start 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- start start 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient: init 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe798095a50 0x7fe798141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe798095a50 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 0x7fe7981418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe798095e20 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798131cd0 0x7fe79811e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): picked mon.noname-c con 0x7fe798131cd0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7980a2ac0 con 0x7fe798095e20 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe798120920 con 0x7fe798095a50 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe798120b20 con 0x7fe798131cd0 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79759e700 10 monclient(hunting): authenticate will time out at 1070.399641s 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe796d9d700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798131cd0 msgr2=0x7fe79811e620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe796d9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798131cd0 0x7fe79811e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:49.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 0x7fe7981418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 0x7fe7981418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39414/0 (socket says 172.21.15.104:39414) 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 1 -- 172.21.15.104:0/4209217472 learned_addr learned my addr 172.21.15.104:0/4209217472 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 10 monclient(hunting): get_auth_request con 0x7fe798095e20 auth_method 0 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe79659c700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe798095a50 0x7fe798141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.403+0000 7fe78ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79659c700 10 monclient(hunting): get_auth_request con 0x7fe798095a50 auth_method 0 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:49.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 10 monclient(hunting): handle_auth_done global_id 14706 payload 824 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 -- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798131cd0 msgr2=0x7fe79811e620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798131cd0 0x7fe79811e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 -- 172.21.15.104:0/4209217472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe798095a50 msgr2=0x7fe798141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe798095a50 0x7fe798141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 10 monclient: _finish_hunting 0 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 monclient: found mon.noname-a 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:49.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 -- 172.21.15.104:0/4209217472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe780009420 con 0x7fe798095e20 2024-09-18T14:25:49.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 -- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 msgr2=0x7fe7981418a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78ffff700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 0x7fe7981418a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe78000ecb0 tx=0x7fe7800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 1 -- 172.21.15.104:0/4209217472 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78000cb60 con 0x7fe798095e20 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: dump: 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.425 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: _finish_auth 0 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 1 -- 172.21.15.104:0/4209217472 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78000ccc0 con 0x7fe798095e20 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79759e700 5 monclient: authenticate success, global_id 14706 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79759e700 10 monclient: _renew_subs 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe790009ae0 con 0x7fe798095e20 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 1 -- 172.21.15.104:0/4209217472 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780013470 con 0x7fe798095e20 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe78e7fc700 10 monclient: dump: 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:49.427 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79559a700 4 set_mon_vals no callback set 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79759e700 10 monclient: _renew_subs 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.407+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe798121ec0 con 0x7fe798095e20 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe78e7fc700 1 -- 172.21.15.104:0/4209217472 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fe780013820 con 0x7fe798095e20 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe79759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe79759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe79759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7980633b0 con 0x7fe798095e20 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe78e7fc700 1 --2- 172.21.15.104:0/4209217472 >> v2:172.21.15.174:6812/17603 conn(0x7fe778035a30 0x7fe778037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe78e7fc700 1 -- 172.21.15.104:0/4209217472 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe780048200 con 0x7fe798095e20 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe79659c700 1 --2- 172.21.15.104:0/4209217472 >> v2:172.21.15.174:6812/17603 conn(0x7fe778035a30 0x7fe778037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:49.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe79659c700 10 monclient: get_auth_request con 0x7fe778035a30 auth_method 0 2024-09-18T14:25:49.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.411+0000 7fe79659c700 1 --2- 172.21.15.104:0/4209217472 >> v2:172.21.15.174:6812/17603 conn(0x7fe778035a30 0x7fe778037ee0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe7880038e0 tx=0x7fe788009120 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:49.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.419+0000 7fe78e7fc700 1 -- 172.21.15.104:0/4209217472 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe7800234d0 con 0x7fe798095e20 2024-09-18T14:25:49.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.419+0000 7fe78e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:49.429 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.419+0000 7fe78e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:49.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:49.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:49.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:49.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe798068a50 con 0x7fe798095e20 2024-09-18T14:25:49.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe78e7fc700 1 -- 172.21.15.104:0/4209217472 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fe7800232f0 con 0x7fe798095e20 2024-09-18T14:25:49.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe78e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:49.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe78e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:49.646 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 >> v2:172.21.15.174:6812/17603 conn(0x7fe778035a30 msgr2=0x7fe778037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 --2- 172.21.15.104:0/4209217472 >> v2:172.21.15.174:6812/17603 conn(0x7fe778035a30 0x7fe778037ee0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe7880038e0 tx=0x7fe788009120 comp rx=0 tx=0).stop 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 10 monclient: shutdown 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 msgr2=0x7fe7981418a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 0x7fe7981418a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe78000ecb0 tx=0x7fe7800091a0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 shutdown_connections 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 --2- 172.21.15.104:0/4209217472 >> v2:172.21.15.174:6812/17603 conn(0x7fe778035a30 0x7fe778037ee0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe798131cd0 0x7fe79811e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe798095e20 0x7fe7981418a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 --2- 172.21.15.104:0/4209217472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe798095a50 0x7fe798141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:49.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 >> 172.21.15.104:0/4209217472 conn(0x7fe7980a3120 msgr2=0x7fe7980a10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:49.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 shutdown_connections 2024-09-18T14:25:49.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:49.639+0000 7fe79759e700 1 -- 172.21.15.104:0/4209217472 wait complete. 2024-09-18T14:25:52.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:52.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.771+0000 7fe74f77a700 10 monclient: get_monmap_and_config 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.771+0000 7fe74f77a700 10 monclient: build_initial_monmap 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.771+0000 7fe74f77a700 10 monclient: monmap: 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:52.773242+0000 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:52.773242+0000 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:52.774 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:52.775 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:52.775 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:52.775 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:52.775 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:52.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.771+0000 7fe74f77a700 1 Processor -- start 2024-09-18T14:25:52.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.771+0000 7fe74f77a700 1 -- start start 2024-09-18T14:25:52.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.771+0000 7fe74f77a700 10 monclient: init 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 0x7fe74806b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient(hunting): picked mon.noname-b con 0x7fe7480a23e0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe74806ee90 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7481342d0 addr v1:172.21.15.104:6789/0 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74806d260 con 0x7fe7481342d0 2024-09-18T14:25:52.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74d516700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 msgr2=0x7fe74806b880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7480a27b0 con 0x7fe74806ee90 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74806d820 con 0x7fe7480a23e0 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74d516700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 0x7fe74806b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 10 monclient(hunting): authenticate will time out at 1073.764699s 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74dd17700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe7481342d0 0x7fe7481366b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36312/0 (socket says 172.21.15.104:36312) 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74dd17700 1 -- 172.21.15.104:0/3458661097 learned_addr learned my addr 172.21.15.104:0/3458661097 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:25:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74dd17700 1 -- 172.21.15.104:0/3458661097 >> v1:172.21.15.104:6789/0 conn(0x7fe7481342d0 legacy=0x7fe7481366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593469275 0 0) 0x7fe74806d260 con 0x7fe7481342d0 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 10 monclient(hunting): my global_id is 14709 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe734003570 con 0x7fe7481342d0 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74cd15700 1 -- 172.21.15.104:0/3458661097 >> v1:172.21.15.174:6789/0 conn(0x7fe74806ee90 legacy=0x7fe74806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799679672 0 0) 0x7fe734003570 con 0x7fe7481342d0 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 msgr2=0x7fe74806b880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 0x7fe74806b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 >> v1:172.21.15.174:6789/0 conn(0x7fe74806ee90 legacy=0x7fe74806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 10 monclient: _finish_hunting 0 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 monclient: found mon.noname-a-legacy 2024-09-18T14:25:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.104:6789/0 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74806cf80 con 0x7fe7481342d0 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 10 monclient: _finish_auth 0 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 5 monclient: authenticate success, global_id 14709 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe74f77a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7fe740001bd0 con 0x7fe7481342d0 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.775+0000 7fe73ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: dump: 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 >> v1:172.21.15.104:6789/0 conn(0x7fe7481342d0 legacy=0x7fe7481366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:25:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 0x7fe734003ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): picked mon.c con 0x7fe7480a23e0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734004210 0x7fe734004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): picked mon.b con 0x7fe734004210 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe734004f50 0x7fe73400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74d516700 1 -- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 msgr2=0x7fe734003ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): picked mon.a con 0x7fe734004f50 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74806d820 con 0x7fe734004f50 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73400bfa0 con 0x7fe734004210 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74d516700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 0x7fe734003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73400c0e0 con 0x7fe7480a23e0 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe734004f50 0x7fe73400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): get_auth_request con 0x7fe734004f50 auth_method 0 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74f77a700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74cd15700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734004210 0x7fe734004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74cd15700 10 monclient(hunting): get_auth_request con 0x7fe734004210 auth_method 0 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74cd15700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74cd15700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74cd15700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient(hunting): handle_auth_done global_id 14709 payload 293 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 -- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 msgr2=0x7fe734003ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe7480a23e0 0x7fe734003ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 -- 172.21.15.104:0/3458661097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734004210 msgr2=0x7fe734004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734004210 0x7fe734004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient: _finish_hunting 0 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 monclient: found mon.a 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:52.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 -- 172.21.15.104:0/3458661097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe734003570 con 0x7fe734004f50 2024-09-18T14:25:52.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 10 monclient: _finish_auth 0 2024-09-18T14:25:52.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:52.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 -- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe734004f50 msgr2=0x7fe73400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:52.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe734004f50 0x7fe73400ba60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe74000abb0 tx=0x7fe74000a260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23846dd8087df02b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:52.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74dd17700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe734004f50 0x7fe73400ba60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe74000abb0 tx=0x7fe74000a260 comp rx=0 tx=0).queue_received will delay after 1.72667e+09 on 0x7fe74000c5d0 mon_map magic: 0 v1 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74000c5d0 con 0x7fe734004f50 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: dump: 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74f77a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74000c730 con 0x7fe734004f50 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe73ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74f77a700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.779+0000 7fe74f77a700 4 set_mon_vals no callback set 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient: shutdown 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe734004f50 msgr2=0x7fe73400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 --2- 172.21.15.104:0/3458661097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe734004f50 0x7fe73400ba60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe74000abb0 tx=0x7fe74000a260 comp rx=0 tx=0).stop 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74dd17700 1 -- 172.21.15.104:0/3458661097 reap_dead start 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe73ffff700 1 -- 172.21.15.104:0/3458661097 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7fe74000caf0 con 0x7fe734004f50 2024-09-18T14:25:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe73ffff700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- 172.21.15.104:0/3458661097 shutdown_connections 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- 172.21.15.104:0/3458661097 >> 172.21.15.104:0/3458661097 conn(0x7fe748067e50 msgr2=0x7fe7480a1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- 172.21.15.104:0/3458661097 shutdown_connections 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- 172.21.15.104:0/3458661097 wait complete. 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient: build_initial_monmap 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient: monmap: 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 Processor -- start 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- start start 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient: init 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe734004f50 0x7fe74809c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): picked mon.noname-c con 0x7fe734004f50 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 0x7fe74811dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): picked mon.noname-a con 0x7fe74811dad0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe74811e3e0 0x7fe748145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): picked mon.noname-b con 0x7fe74811e3e0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74809afe0 con 0x7fe74811dad0 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74d516700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe734004f50 msgr2=0x7fe74809c540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe748120280 con 0x7fe74811e3e0 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74811f1c0 con 0x7fe734004f50 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74f77a700 10 monclient(hunting): authenticate will time out at 1073.775960s 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.783+0000 7fe74d516700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe734004f50 0x7fe74809c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 0x7fe74811dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 0x7fe74811dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48106/0 (socket says 172.21.15.104:48106) 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 -- 172.21.15.104:0/3210162073 learned_addr learned my addr 172.21.15.104:0/3210162073 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): get_auth_request con 0x7fe74811dad0 auth_method 0 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe74811e3e0 0x7fe748145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 10 monclient(hunting): get_auth_request con 0x7fe74811e3e0 auth_method 0 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74dd17700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient(hunting): handle_auth_done global_id 14712 payload 824 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 -- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe734004f50 msgr2=0x7fe74809c540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe734004f50 0x7fe74809c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 -- 172.21.15.104:0/3210162073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe74811e3e0 msgr2=0x7fe748145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe74811e3e0 0x7fe748145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient: _finish_hunting 0 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 monclient: found mon.noname-a 2024-09-18T14:25:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 -- 172.21.15.104:0/3210162073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74809cf60 con 0x7fe74811dad0 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 -- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 msgr2=0x7fe74811dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74cd15700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 0x7fe74811dea0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe73800e580 tx=0x7fe738009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 1 -- 172.21.15.104:0/3210162073 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7380074f0 con 0x7fe74811dad0 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: dump: 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: _finish_auth 0 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 1 -- 172.21.15.104:0/3210162073 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe738010040 con 0x7fe74811dad0 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74f77a700 5 monclient: authenticate success, global_id 14712 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74f77a700 10 monclient: _renew_subs 2024-09-18T14:25:52.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74811ffa0 con 0x7fe74811dad0 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 1 -- 172.21.15.104:0/3210162073 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738005a40 con 0x7fe74811dad0 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73d7fa700 10 monclient: dump: 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:52.803 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74f77a700 10 monclient: _renew_subs 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe73f7fe700 4 set_mon_vals no callback set 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.787+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74811e810 con 0x7fe74811dad0 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe73d7fa700 1 -- 172.21.15.104:0/3210162073 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fe7380143f0 con 0x7fe74811dad0 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe74f77a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe74f77a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe74f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7480636d0 con 0x7fe74811dad0 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe73d7fa700 1 --2- 172.21.15.104:0/3210162073 >> v2:172.21.15.174:6812/17603 conn(0x7fe72c035a80 0x7fe72c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe73d7fa700 1 -- 172.21.15.104:0/3210162073 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe738046d80 con 0x7fe74811dad0 2024-09-18T14:25:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe74d516700 1 --2- 172.21.15.104:0/3210162073 >> v2:172.21.15.174:6812/17603 conn(0x7fe72c035a80 0x7fe72c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.791+0000 7fe74d516700 10 monclient: get_auth_request con 0x7fe72c035a80 auth_method 0 2024-09-18T14:25:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.795+0000 7fe74d516700 1 --2- 172.21.15.104:0/3210162073 >> v2:172.21.15.174:6812/17603 conn(0x7fe72c035a80 0x7fe72c037f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe744006fd0 tx=0x7fe744006e90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.799+0000 7fe73d7fa700 1 -- 172.21.15.104:0/3210162073 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe738005db0 con 0x7fe74811dad0 2024-09-18T14:25:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.799+0000 7fe73d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:52.799+0000 7fe73d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:53.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.019+0000 7fe74f77a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:53.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.019+0000 7fe74f77a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:53.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.019+0000 7fe74f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:53.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.019+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe748069620 con 0x7fe74811dad0 2024-09-18T14:25:53.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.019+0000 7fe73d7fa700 1 -- 172.21.15.104:0/3210162073 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fe738019020 con 0x7fe74811dad0 2024-09-18T14:25:53.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.019+0000 7fe73d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:53.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.019+0000 7fe73d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:53.022 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 >> v2:172.21.15.174:6812/17603 conn(0x7fe72c035a80 msgr2=0x7fe72c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 --2- 172.21.15.104:0/3210162073 >> v2:172.21.15.174:6812/17603 conn(0x7fe72c035a80 0x7fe72c037f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe744006fd0 tx=0x7fe744006e90 comp rx=0 tx=0).stop 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 10 monclient: shutdown 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 msgr2=0x7fe74811dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 0x7fe74811dea0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe73800e580 tx=0x7fe738009040 comp rx=0 tx=0).stop 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 shutdown_connections 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 --2- 172.21.15.104:0/3210162073 >> v2:172.21.15.174:6812/17603 conn(0x7fe72c035a80 0x7fe72c037f30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe74811e3e0 0x7fe748145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe74811dad0 0x7fe74811dea0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:53.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 --2- 172.21.15.104:0/3210162073 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fe734004f50 0x7fe74809c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:53.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 >> 172.21.15.104:0/3210162073 conn(0x7fe748067e50 msgr2=0x7fe74809a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:53.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 shutdown_connections 2024-09-18T14:25:53.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:53.023+0000 7fe74f77a700 1 -- 172.21.15.104:0/3210162073 wait complete. 2024-09-18T14:25:56.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.143+0000 7f351359e700 10 monclient: get_monmap_and_config 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.143+0000 7f351359e700 10 monclient: build_initial_monmap 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.143+0000 7f351359e700 10 monclient: monmap: 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:25:56.146318+0000 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:25:56.146318+0000 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:56.147 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:56.148 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:25:56.148 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:25:56.148 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:25:56.148 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:25:56.148 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:25:56.148 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:56.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.143+0000 7f351359e700 1 Processor -- start 2024-09-18T14:25:56.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.143+0000 7f351359e700 1 -- start start 2024-09-18T14:25:56.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.143+0000 7f351359e700 10 monclient: init 2024-09-18T14:25:56.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:56.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T14:25:56.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f35140a59d0 0x7f351409ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:56.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient(hunting): picked mon.noname-b con 0x7f35140a59d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f351409bea0 addr v1:172.21.15.174:6789/0 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f351409f240 0x7f35140a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient(hunting): picked mon.noname-c con 0x7f351409f240 addr v2:172.21.15.174:3300/0 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351409af10 con 0x7f351409bea0 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35140a1870 con 0x7f351409f240 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35140a5da0 con 0x7f35140a59d0 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351359e700 10 monclient(hunting): authenticate will time out at 1077.136577s 2024-09-18T14:25:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351259c700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f35140a59d0 msgr2=0x7f351409ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f351259c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f35140a59d0 0x7f351409ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f351409f240 0x7f35140a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f351409f240 0x7f35140a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.104:51968/0 (socket says 172.21.15.104:51968) 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 -- 172.21.15.104:0/2380607138 learned_addr learned my addr 172.21.15.104:0/2380607138 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): get_auth_request con 0x7f351409f240 auth_method 0 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3511d9b700 1 --1- 172.21.15.104:0/2380607138 >> v1:172.21.15.174:6789/0 conn(0x7f351409bea0 0x7f351409fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.104:43046/0 (socket says 172.21.15.104:43046) 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:56.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient(hunting): handle_auth_done global_id 5410 payload 274 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 -- 172.21.15.104:0/2380607138 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f35140a59d0 msgr2=0x7f351409ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 --2- 172.21.15.104:0/2380607138 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f35140a59d0 0x7f351409ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 -- 172.21.15.104:0/2380607138 >> v1:172.21.15.174:6789/0 conn(0x7f351409bea0 legacy=0x7f351409fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 monclient: found mon.noname-c 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:25:56.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.147+0000 7f3512d9d700 1 -- 172.21.15.104:0/2380607138 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351409cae0 con 0x7f351409f240 2024-09-18T14:25:56.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 1 -- 172.21.15.104:0/2380607138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409f240 msgr2=0x7f35140a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:56.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 1 --2- 172.21.15.104:0/2380607138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409f240 0x7f35140a03c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f351409b980 tx=0x7f350c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74decac04a28dd45 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:56.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 1 -- 172.21.15.104:0/2380607138 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00e070 con 0x7f351409f240 2024-09-18T14:25:56.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: dump: 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:56.160 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: _finish_auth 0 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 5 monclient: authenticate success, global_id 5410 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 1 -- 172.21.15.104:0/2380607138 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c00bc60 con 0x7f351409f240 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient: get_monmap_and_config success 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 4 set_mon_vals no callback set 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 1 -- 172.21.15.104:0/2380607138 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c007400 con 0x7f351409f240 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351159a700 10 monclient: dump: 2024-09-18T14:25:56.161 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient: shutdown 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- 172.21.15.104:0/2380607138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409f240 msgr2=0x7f35140a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 --2- 172.21.15.104:0/2380607138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409f240 0x7f35140a03c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f351409b980 tx=0x7f350c009840 comp rx=0 tx=0).stop 2024-09-18T14:25:56.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- 172.21.15.104:0/2380607138 shutdown_connections 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 --2- 172.21.15.104:0/2380607138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409f240 0x7f35140a03c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 --2- 172.21.15.104:0/2380607138 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f35140a59d0 0x7f351409ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- 172.21.15.104:0/2380607138 >> 172.21.15.104:0/2380607138 conn(0x7f3514067c10 msgr2=0x7f35140a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- 172.21.15.104:0/2380607138 shutdown_connections 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- 172.21.15.104:0/2380607138 wait complete. 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient: build_initial_monmap 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient: monmap: 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:25:56.163 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 Processor -- start 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- start start 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient: init 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient: _reopen_session rank -1 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409bea0 0x7f3514131b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): picked mon.noname-b con 0x7f351409bea0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f351409f240 0x7f351411b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:56.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): picked mon.noname-c con 0x7f351409f240 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 0x7f351411be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): picked mon.noname-a con 0x7f35140a59d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351406f430 con 0x7f35140a59d0 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35141328e0 con 0x7f351409bea0 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3514141e90 con 0x7f351409f240 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): _renew_subs 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351359e700 10 monclient(hunting): authenticate will time out at 1077.143296s 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3511d9b700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f351409f240 msgr2=0x7f351411b920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3511d9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f351409f240 0x7f351411b920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:25:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 0x7f351411be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 0x7f351411be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48120/0 (socket says 172.21.15.104:48120) 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 1 -- 172.21.15.104:0/2087853845 learned_addr learned my addr 172.21.15.104:0/2087853845 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): get_auth_request con 0x7f35140a59d0 auth_method 0 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351259c700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409bea0 0x7f3514131b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351259c700 10 monclient(hunting): get_auth_request con 0x7f351409bea0 auth_method 0 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:25:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f351259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.151+0000 7f3512d9d700 10 monclient(hunting): handle_auth_done global_id 14715 payload 824 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 -- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f351409f240 msgr2=0x7f351411b920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f351409f240 0x7f351411b920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 -- 172.21.15.104:0/2087853845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409bea0 msgr2=0x7f3514131b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409bea0 0x7f3514131b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 monclient: found mon.noname-a 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 -- 172.21.15.104:0/2087853845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c009ae0 con 0x7f35140a59d0 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351259c700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409bea0 0x7f3514131b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:25:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 -- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 msgr2=0x7f351411be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 0x7f351411be60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35140a62c0 tx=0x7f350c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3512d9d700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 0x7f351411be60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35140a62c0 tx=0x7f350c018000 comp rx=0 tx=0).queue_received will delay after 1.72667e+09 on 0x7f350c026460 mon_map magic: 0 v1 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 1 -- 172.21.15.104:0/2087853845 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c026460 con 0x7f35140a59d0 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 10 monclient: dump: 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:56.168 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 10 monclient: _finish_auth 0 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351359e700 5 monclient: authenticate success, global_id 14715 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351359e700 10 monclient: _renew_subs 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 1 -- 172.21.15.104:0/2087853845 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c0265c0 con 0x7f35140a59d0 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3514134b90 con 0x7f35140a59d0 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f35027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f3510d99700 4 set_mon_vals no callback set 2024-09-18T14:25:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351359e700 10 monclient: _renew_subs 2024-09-18T14:25:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.155+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351411d130 con 0x7f35140a59d0 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f35027fc700 1 -- 172.21.15.104:0/2087853845 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c026980 con 0x7f35140a59d0 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f35027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f35027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f35027fc700 10 monclient: dump: 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f35027fc700 1 -- 172.21.15.104:0/2087853845 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f350c003040 con 0x7f35140a59d0 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f35027fc700 1 --2- 172.21.15.104:0/2087853845 >> v2:172.21.15.174:6812/17603 conn(0x7f34f4035a80 0x7f34f4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f351359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f351359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f351359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35140633b0 con 0x7f35140a59d0 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f351259c700 1 --2- 172.21.15.104:0/2087853845 >> v2:172.21.15.174:6812/17603 conn(0x7f34f4035a80 0x7f34f4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:25:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f351259c700 10 monclient: get_auth_request con 0x7f34f4035a80 auth_method 0 2024-09-18T14:25:56.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.159+0000 7f351259c700 1 --2- 172.21.15.104:0/2087853845 >> v2:172.21.15.174:6812/17603 conn(0x7f34f4035a80 0x7f34f4037f30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3504004fd0 tx=0x7f3504004aa0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:25:56.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.167+0000 7f35027fc700 1 -- 172.21.15.104:0/2087853845 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f350c049e80 con 0x7f35140a59d0 2024-09-18T14:25:56.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.779+0000 7f35027fc700 1 -- 172.21.15.104:0/2087853845 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f350c021030 con 0x7f35140a59d0 2024-09-18T14:25:56.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.779+0000 7f35027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:25:56.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:56.779+0000 7f35027fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:25:57.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.007+0000 7f351359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:25:57.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.007+0000 7f351359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:57.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.007+0000 7f351359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:25:57.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.007+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f351411dd60 con 0x7f35140a59d0 2024-09-18T14:25:57.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.007+0000 7f35027fc700 1 -- 172.21.15.104:0/2087853845 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f350c021030 con 0x7f35140a59d0 2024-09-18T14:25:57.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.007+0000 7f35027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:25:57.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.007+0000 7f35027fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:25:57.011 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 >> v2:172.21.15.174:6812/17603 conn(0x7f34f4035a80 msgr2=0x7f34f4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 --2- 172.21.15.104:0/2087853845 >> v2:172.21.15.174:6812/17603 conn(0x7f34f4035a80 0x7f34f4037f30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3504004fd0 tx=0x7f3504004aa0 comp rx=0 tx=0).stop 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 10 monclient: shutdown 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 msgr2=0x7f351411be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 0x7f351411be60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35140a62c0 tx=0x7f350c018000 comp rx=0 tx=0).stop 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 shutdown_connections 2024-09-18T14:25:57.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 --2- 172.21.15.104:0/2087853845 >> v2:172.21.15.174:6812/17603 conn(0x7f34f4035a80 0x7f34f4037f30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:57.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35140a59d0 0x7f351411be60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:57.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f351409f240 0x7f351411b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:57.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 --2- 172.21.15.104:0/2087853845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f351409bea0 0x7f3514131b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:25:57.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 >> 172.21.15.104:0/2087853845 conn(0x7f3514067c10 msgr2=0x7f3514134da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:25:57.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 shutdown_connections 2024-09-18T14:25:57.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:25:57.011+0000 7f351359e700 1 -- 172.21.15.104:0/2087853845 wait complete. 2024-09-18T14:26:00.024 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:26:00.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient: get_monmap_and_config 2024-09-18T14:26:00.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient: build_initial_monmap 2024-09-18T14:26:00.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient: monmap: 2024-09-18T14:26:00.133 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:26:00.133 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:00.133 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:26:00.132573+0000 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:26:00.132573+0000 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:26:00.134 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:00.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 1 Processor -- start 2024-09-18T14:26:00.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 1 -- start start 2024-09-18T14:26:00.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient: init 2024-09-18T14:26:00.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:00.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa1400a59d0 addr v2:172.21.15.174:3300/0 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa14009bea0 addr v1:172.21.15.174:6789/0 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa14009f240 addr v2:172.21.15.104:3300/0 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14009af10 con 0x7fa14009bea0 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1400a1870 con 0x7fa14009f240 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1400a5da0 con 0x7fa1400a59d0 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13f59e700 10 monclient(hunting): authenticate will time out at 1081.122144s 2024-09-18T14:26:00.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41560/0 (socket says 172.21.15.104:41560) 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 learned_addr learned my addr 172.21.15.104:0/927658144 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request con 0x7fa14009f240 auth_method 0 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 10 monclient(hunting): get_auth_request con 0x7fa1400a59d0 auth_method 0 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:00.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:00.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:00.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:26:00.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:00.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:00.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.131+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:00.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.135+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> v1:172.21.15.174:6789/0 conn(0x7fa14009bea0 legacy=0x7fa14009fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:00.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:00.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request con 0x7fa14009f240 auth_method 0 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 10 monclient(hunting): get_auth_request con 0x7fa1400a59d0 auth_method 0 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:00.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:26:00.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:26:00.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:00.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:00.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.335+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:26:00.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:00.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request con 0x7fa14009f240 auth_method 0 2024-09-18T14:26:00.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 10 monclient(hunting): get_auth_request con 0x7fa1400a59d0 auth_method 0 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:26:00.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:00.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:00.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:00.739+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:26:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:01.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request con 0x7fa14009f240 auth_method 0 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 10 monclient(hunting): get_auth_request con 0x7fa1400a59d0 auth_method 0 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:26:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:26:01.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:01.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:01.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:01.543+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:26:03.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 10 monclient(hunting): tick 2024-09-18T14:26:03.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 monclient(hunting): continuing hunt 2024-09-18T14:26:03.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T14:26:03.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 msgr2=0x7fa14009ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:26:03.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.174:3300/0 conn(0x7fa1400a59d0 0x7fa14009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 -- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 msgr2=0x7fa1400a03c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 --2- 172.21.15.104:0/927658144 >> v2:172.21.15.104:3300/0 conn(0x7fa14009f240 0x7fa1400a03c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 -- 172.21.15.104:0/927658144 >> v1:172.21.15.174:6789/0 conn(0x7fa14009bea0 legacy=0x7fa14009fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 10 monclient: _add_conns ranks=[2,0] 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 0x7fa120005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa120003740 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1200062b0 addr v1:172.21.15.104:6789/0 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 -- 172.21.15.104:0/927658144 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14009cae0 con 0x7fa1200062b0 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1400a1870 con 0x7fa120003740 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa137fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 msgr2=0x7fa120005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:03.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.131+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 0x7fa120005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:03.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.135+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> v1:172.21.15.104:6789/0 conn(0x7fa1200062b0 legacy=0x7fa120008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:03.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.335+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 msgr2=0x7fa120005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:03.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.335+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 0x7fa120005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:26:03.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.739+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 msgr2=0x7fa120005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:03.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:03.739+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 0x7fa120005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:26:04.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549975863 0 0) 0x7fa14009cae0 con 0x7fa1200062b0 2024-09-18T14:26:04.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 10 monclient(hunting): my global_id is 14718 2024-09-18T14:26:04.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12c003570 con 0x7fa1200062b0 2024-09-18T14:26:04.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2327449163 0 0) 0x7fa12c003570 con 0x7fa1200062b0 2024-09-18T14:26:04.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 msgr2=0x7fa120005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:26:04.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa120003740 0x7fa120005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.015 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 1 monclient: found mon.noname-a-legacy 2024-09-18T14:26:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.104:6789/0 2024-09-18T14:26:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12000b0f0 con 0x7fa1200062b0 2024-09-18T14:26:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.011+0000 7fa13d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7fa128001800 con 0x7fa1200062b0 2024-09-18T14:26:04.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T14:26:04.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient: dump: 2024-09-18T14:26:04.023 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.024 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.024 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.024 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.024 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.025 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.025 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.025 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.025 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.025 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-18T14:26:04.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:04.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 >> v1:172.21.15.104:6789/0 conn(0x7fa1200062b0 legacy=0x7fa120008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:26:04.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 reap_dead start 2024-09-18T14:26:04.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 0x7fa12c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): picked mon.c con 0x7fa12c003a60 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:26:04.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13e59c700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 msgr2=0x7fa12c003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:04.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 0x7fa12c003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:04.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): picked mon.b con 0x7fa12c004810 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:26:04.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): picked mon.a con 0x7fa12c0053c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:26:04.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1400a1870 con 0x7fa12c0053c0 2024-09-18T14:26:04.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c00c320 con 0x7fa12c004810 2024-09-18T14:26:04.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c00c4e0 con 0x7fa12c003a60 2024-09-18T14:26:04.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:04.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13d59a700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:26:04.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): get_auth_request con 0x7fa12c004810 auth_method 0 2024-09-18T14:26:04.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:04.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:26:04.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request con 0x7fa12c0053c0 auth_method 0 2024-09-18T14:26:04.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:04.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_done global_id 14718 payload 293 2024-09-18T14:26:04.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 msgr2=0x7fa12c003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:26:04.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 0x7fa12c003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 msgr2=0x7fa12c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:26:04.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 monclient: found mon.b 2024-09-18T14:26:04.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.174:3300/0 2024-09-18T14:26:04.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c003570 con 0x7fa12c004810 2024-09-18T14:26:04.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13f59e700 5 monclient: authenticate success, global_id 14718 2024-09-18T14:26:04.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:26:04.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:26:04.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 msgr2=0x7fa12c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:26:04.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 msgr2=0x7fa12c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:26:04.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fa14009b0a0 tx=0x7fa128007940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:26:04.035 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.015+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:04.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient: get_auth_request con 0x7fa12c004810 auth_method 0 2024-09-18T14:26:04.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 0 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:26:04.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient: ms_handle_reset current mon [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:26:04.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:04.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 msgr2=0x7fa12c004c80 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T14:26:04.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c004810 0x7fa12c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:26:04.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1200062b0 0x7fa12c00eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): picked mon.b con 0x7fa1200062b0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:26:04.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): picked mon.a con 0x7fa12c0053c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:26:04.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 0x7fa12c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): picked mon.c con 0x7fa12c003a60 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c00f4a0 con 0x7fa12c0053c0 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c00f5e0 con 0x7fa1200062b0 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c00f720 con 0x7fa12c003a60 2024-09-18T14:26:04.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:04.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13d59a700 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:26:04.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13ed9d700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 msgr2=0x7fa12c003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:04.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 0x7fa12c003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:04.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): get_auth_request con 0x7fa12c0053c0 auth_method 0 2024-09-18T14:26:04.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:04.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1200062b0 0x7fa12c00eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 10 monclient(hunting): get_auth_request con 0x7fa1200062b0 auth_method 0 2024-09-18T14:26:04.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:26:04.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:26:04.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13e59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:26:04.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_done global_id 14718 payload 293 2024-09-18T14:26:04.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 msgr2=0x7fa12c003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 0x7fa12c003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1200062b0 msgr2=0x7fa12c00eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1200062b0 0x7fa12c00eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 monclient: found mon.a 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fa14009cae0 con 0x7fa12c0053c0 2024-09-18T14:26:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.219+0000 7fa13dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13dd9b700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 msgr2=0x7fa12c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c4b0 tx=0x7fa128007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c722c6cc5bd0dc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128019070 con 0x7fa12c0053c0 2024-09-18T14:26:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13d59a700 10 monclient: dump: 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.235 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13d59a700 1 -- 172.21.15.104:0/927658144 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12800ccd0 con 0x7fa12c0053c0 2024-09-18T14:26:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:26:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:26:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 10 monclient: get_monmap_and_config success 2024-09-18T14:26:04.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 4 set_mon_vals no callback set 2024-09-18T14:26:04.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 10 monclient: shutdown 2024-09-18T14:26:04.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:26:04.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 -- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 msgr2=0x7fa12c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c4b0 tx=0x7fa128007940 comp rx=0 tx=0).stop 2024-09-18T14:26:04.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 -- 172.21.15.104:0/927658144 shutdown_connections 2024-09-18T14:26:04.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa12c003a60 0x7fa12c003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c0053c0 0x7fa12c00ba60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 --2- 172.21.15.104:0/927658144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1200062b0 0x7fa12c00eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 -- 172.21.15.104:0/927658144 >> 172.21.15.104:0/927658144 conn(0x7fa140067c10 msgr2=0x7fa1400a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:26:04.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 -- 172.21.15.104:0/927658144 shutdown_connections 2024-09-18T14:26:04.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 1 -- 172.21.15.104:0/927658144 wait complete. 2024-09-18T14:26:04.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 10 monclient: build_initial_monmap 2024-09-18T14:26:04.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.223+0000 7fa13f59e700 10 monclient: monmap: 2024-09-18T14:26:04.246 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:26:04.246 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:26:04.246 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 Processor -- start 2024-09-18T14:26:04.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 -- start start 2024-09-18T14:26:04.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient: init 2024-09-18T14:26:04.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:04.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:26:04.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1200062b0 0x7fa14013d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa1200062b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:26:04.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 0x7fa14013cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa12c003a60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:26:04.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0053c0 0x7fa140140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa12c0053c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:26:04.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1401224a0 con 0x7fa12c003a60 2024-09-18T14:26:04.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa140121ee0 con 0x7fa12c0053c0 2024-09-18T14:26:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14011f9c0 con 0x7fa1200062b0 2024-09-18T14:26:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13f59e700 10 monclient(hunting): authenticate will time out at 1085.217913s 2024-09-18T14:26:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13e59c700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1200062b0 msgr2=0x7fa14013d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13e59c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1200062b0 0x7fa14013d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:04.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 0x7fa14013cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 0x7fa14013cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41614/0 (socket says 172.21.15.104:41614) 2024-09-18T14:26:04.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 -- 172.21.15.104:0/1014273681 learned_addr learned my addr 172.21.15.104:0/1014273681 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:26:04.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): get_auth_request con 0x7fa12c003a60 auth_method 0 2024-09-18T14:26:04.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0053c0 0x7fa140140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request con 0x7fa12c0053c0 auth_method 0 2024-09-18T14:26:04.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient(hunting): handle_auth_done global_id 14721 payload 824 2024-09-18T14:26:04.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 -- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1200062b0 msgr2=0x7fa14013d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:26:04.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1200062b0 0x7fa14013d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 -- 172.21.15.104:0/1014273681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0053c0 msgr2=0x7fa140140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0053c0 0x7fa140140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 monclient: found mon.noname-a 2024-09-18T14:26:04.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.227+0000 7fa13dd9b700 1 -- 172.21.15.104:0/1014273681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140121c00 con 0x7fa12c003a60 2024-09-18T14:26:04.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13dd9b700 1 -- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 msgr2=0x7fa14013cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13dd9b700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 0x7fa14013cb60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa128000c00 tx=0x7fa128016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 1 -- 172.21.15.104:0/1014273681 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128025070 con 0x7fa12c003a60 2024-09-18T14:26:04.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:26:04.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: dump: 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.256 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13f59e700 5 monclient: authenticate success, global_id 14721 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 1 -- 172.21.15.104:0/1014273681 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12800ce40 con 0x7fa12c003a60 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:26:04.257 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 1 -- 172.21.15.104:0/1014273681 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128005c70 con 0x7fa12c003a60 2024-09-18T14:26:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13f59e700 10 monclient: _renew_subs 2024-09-18T14:26:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1401221c0 con 0x7fa12c003a60 2024-09-18T14:26:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13cd99700 4 set_mon_vals no callback set 2024-09-18T14:26:04.258 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 10 monclient: dump: 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.260 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13f59e700 10 monclient: _renew_subs 2024-09-18T14:26:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14013d240 con 0x7fa12c003a60 2024-09-18T14:26:04.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.231+0000 7fa1367fc700 1 -- 172.21.15.104:0/1014273681 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fa128007a80 con 0x7fa12c003a60 2024-09-18T14:26:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa1367fc700 1 --2- 172.21.15.104:0/1014273681 >> v2:172.21.15.174:6812/17603 conn(0x7fa120036860 0x7fa120038d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa1367fc700 1 -- 172.21.15.104:0/1014273681 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fa1280495c0 con 0x7fa12c003a60 2024-09-18T14:26:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa13f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:26:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa13f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:26:04.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa13f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14013d240 con 0x7fa12c003a60 2024-09-18T14:26:04.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa13e59c700 1 --2- 172.21.15.104:0/1014273681 >> v2:172.21.15.174:6812/17603 conn(0x7fa120036860 0x7fa120038d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa13e59c700 10 monclient: get_auth_request con 0x7fa120036860 auth_method 0 2024-09-18T14:26:04.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.235+0000 7fa13e59c700 1 --2- 172.21.15.104:0/1014273681 >> v2:172.21.15.174:6812/17603 conn(0x7fa120036860 0x7fa120038d10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa138009590 tx=0x7fa13800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.243+0000 7fa1367fc700 1 -- 172.21.15.104:0/1014273681 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa12801e410 con 0x7fa12c003a60 2024-09-18T14:26:04.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.243+0000 7fa1367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:26:04.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.243+0000 7fa1367fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:26:04.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa13f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:26:04.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa13f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:26:04.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa13f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa14013d240 con 0x7fa12c003a60 2024-09-18T14:26:04.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa1367fc700 1 -- 172.21.15.104:0/1014273681 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fa128049400 con 0x7fa12c003a60 2024-09-18T14:26:04.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa1367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:26:04.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa1367fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:26:04.466 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 >> v2:172.21.15.174:6812/17603 conn(0x7fa120036860 msgr2=0x7fa120038d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa13f59e700 1 --2- 172.21.15.104:0/1014273681 >> v2:172.21.15.174:6812/17603 conn(0x7fa120036860 0x7fa120038d10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa138009590 tx=0x7fa13800b040 comp rx=0 tx=0).stop 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.463+0000 7fa13f59e700 10 monclient: shutdown 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 msgr2=0x7fa14013cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 0x7fa14013cb60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa128000c00 tx=0x7fa128016000 comp rx=0 tx=0).stop 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 shutdown_connections 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 --2- 172.21.15.104:0/1014273681 >> v2:172.21.15.174:6812/17603 conn(0x7fa120036860 0x7fa120038d10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0053c0 0x7fa140140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c003a60 0x7fa14013cb60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 --2- 172.21.15.104:0/1014273681 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa1200062b0 0x7fa14013d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 >> 172.21.15.104:0/1014273681 conn(0x7fa140067c10 msgr2=0x7fa1400a39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:26:04.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 shutdown_connections 2024-09-18T14:26:04.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.467+0000 7fa13f59e700 1 -- 172.21.15.104:0/1014273681 wait complete. 2024-09-18T14:26:04.480 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T14:26:04.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T14:26:04.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:26:04.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient: get_monmap_and_config 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient: build_initial_monmap 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient: monmap: 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:26:04.585548+0000 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:26:04.585548+0000 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:26:04.592 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 1 Processor -- start 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 1 -- start start 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient: init 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T14:26:04.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ac809d620 addr v1:172.21.15.174:6789/0 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ac812f250 addr v1:172.21.15.104:6789/0 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5ac812f650 0x7f5ac81375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient(hunting): picked mon.noname-a con 0x7f5ac812f650 addr v2:172.21.15.104:3300/0 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80990f0 con 0x7f5ac812f250 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac809d9f0 con 0x7f5ac809d620 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac8096e10 con 0x7f5ac812f650 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acf9ff700 10 monclient(hunting): authenticate will time out at 1085.575479s 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5ac812f650 0x7f5ac81375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5ac812f650 0x7f5ac81375a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41626/0 (socket says 172.21.15.104:41626) 2024-09-18T14:26:04.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 1 -- 172.21.15.104:0/230206253 learned_addr learned my addr 172.21.15.104:0/230206253 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 10 monclient(hunting): get_auth_request con 0x7f5ac812f650 auth_method 0 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.583+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acd79b700 1 -- 172.21.15.104:0/230206253 >> v1:172.21.15.174:6789/0 conn(0x7f5ac809d620 legacy=0x7f5ac812eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5accf9a700 1 -- 172.21.15.104:0/230206253 >> v1:172.21.15.104:6789/0 conn(0x7f5ac812f250 legacy=0x7f5ac8133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_done global_id 14724 payload 274 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 1 -- 172.21.15.104:0/230206253 >> v1:172.21.15.174:6789/0 conn(0x7f5ac809d620 legacy=0x7f5ac812eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 1 -- 172.21.15.104:0/230206253 >> v1:172.21.15.104:6789/0 conn(0x7f5ac812f250 legacy=0x7f5ac8133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 1 monclient: found mon.noname-a 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.587+0000 7f5acdf9c700 1 -- 172.21.15.104:0/230206253 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac809c200 con 0x7f5ac812f650 2024-09-18T14:26:04.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 -- 172.21.15.104:0/230206253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 msgr2=0x7f5ac81375a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.609 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 --2- 172.21.15.104:0/230206253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac81375a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009b10 tx=0x7f5ab8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cbfb78e4a80026 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 1 -- 172.21.15.104:0/230206253 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800bb80 con 0x7f5ac812f650 2024-09-18T14:26:04.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:26:04.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: dump: 2024-09-18T14:26:04.611 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.611 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.611 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.611 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.612 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.612 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.612 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.612 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.612 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.612 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 5 monclient: authenticate success, global_id 14724 2024-09-18T14:26:04.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 1 -- 172.21.15.104:0/230206253 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab800bce0 con 0x7f5ac812f650 2024-09-18T14:26:04.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:26:04.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:26:04.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient: get_monmap_and_config success 2024-09-18T14:26:04.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 1 -- 172.21.15.104:0/230206253 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8011670 con 0x7f5ac812f650 2024-09-18T14:26:04.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 4 set_mon_vals no callback set 2024-09-18T14:26:04.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5ac7fff700 10 monclient: dump: 2024-09-18T14:26:04.615 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.615 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.615 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.615 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient: shutdown 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- 172.21.15.104:0/230206253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 msgr2=0x7f5ac81375a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 --2- 172.21.15.104:0/230206253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac81375a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009b10 tx=0x7f5ab8009250 comp rx=0 tx=0).stop 2024-09-18T14:26:04.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- 172.21.15.104:0/230206253 shutdown_connections 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 --2- 172.21.15.104:0/230206253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac81375a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- 172.21.15.104:0/230206253 >> 172.21.15.104:0/230206253 conn(0x7f5ac80a1290 msgr2=0x7f5ac80a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- 172.21.15.104:0/230206253 shutdown_connections 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- 172.21.15.104:0/230206253 wait complete. 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient: build_initial_monmap 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient: monmap: 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.617 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 Processor -- start 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- start start 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient: init 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac809d620 0x7f5ac81364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): picked mon.noname-b con 0x7f5ac809d620 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f5ac812f250 0x7f5ac806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): picked mon.noname-c con 0x7f5ac812f250 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac806da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): picked mon.noname-a con 0x7f5ac812f650 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8136c20 con 0x7f5ac812f650 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8131bc0 con 0x7f5ac809d620 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac812ffe0 con 0x7f5ac812f250 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acf9ff700 10 monclient(hunting): authenticate will time out at 1085.583188s 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5accf9a700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f5ac812f250 msgr2=0x7f5ac806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac806da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5accf9a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f5ac812f250 0x7f5ac806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac806da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41638/0 (socket says 172.21.15.104:41638) 2024-09-18T14:26:04.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 -- 172.21.15.104:0/887604632 learned_addr learned my addr 172.21.15.104:0/887604632 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): get_auth_request con 0x7f5ac812f650 auth_method 0 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac809d620 0x7f5ac81364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 10 monclient(hunting): get_auth_request con 0x7f5ac809d620 auth_method 0 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient(hunting): handle_auth_done global_id 14727 payload 824 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f5ac812f250 msgr2=0x7f5ac806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f5ac812f250 0x7f5ac806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac809d620 msgr2=0x7f5ac81364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac809d620 0x7f5ac81364e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 monclient: found mon.noname-a 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.591+0000 7f5acdf9c700 1 -- 172.21.15.104:0/887604632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80094f0 con 0x7f5ac812f650 2024-09-18T14:26:04.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acdf9c700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 msgr2=0x7f5ac806da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acdf9c700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac806da80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8068560 tx=0x7f5ab8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab801f070 con 0x7f5ac812f650 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: dump: 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.623 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab800be50 con 0x7f5ac812f650 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 5 monclient: authenticate success, global_id 14727 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 10 monclient: _renew_subs 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac806f370 con 0x7f5ac812f650 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:26:04.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8025670 con 0x7f5ac812f650 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 10 monclient: dump: 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.626 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.626 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.626 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 10 monclient: _renew_subs 2024-09-18T14:26:04.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac8130220 con 0x7f5ac812f650 2024-09-18T14:26:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac77fe700 4 set_mon_vals no callback set 2024-09-18T14:26:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:26:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 0x7f5ac812ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac809b7b0 con 0x7f5ac806dfc0 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f5ab8023070 con 0x7f5ac812f650 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 0x7f5ac812ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 10 monclient: get_auth_request con 0x7f5ac806dfc0 auth_method 0 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 1 --2- 172.21.15.104:0/887604632 >> v2:172.21.15.174:6812/17603 conn(0x7f5ab0035a30 0x7f5ab0037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.595+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f5ab8049710 con 0x7f5ac812f650 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5acd79b700 10 monclient(hunting): handle_auth_done global_id 14730 payload 824 2024-09-18T14:26:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5acd79b700 1 -- 172.21.15.104:0/887604632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ac806e390 con 0x7f5ac806dfc0 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5accf9a700 1 --2- 172.21.15.104:0/887604632 >> v2:172.21.15.174:6812/17603 conn(0x7f5ab0035a30 0x7f5ab0037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5accf9a700 10 monclient: get_auth_request con 0x7f5ab0035a30 auth_method 0 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5acd79b700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 msgr2=0x7f5ac812ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5acd79b700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 0x7f5ac812ce50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f7b0 tx=0x7f5ac0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00074f0 con 0x7f5ac806dfc0 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 10 monclient: dump: 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.630 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5accf9a700 1 --2- 172.21.15.104:0/887604632 >> v2:172.21.15.174:6812/17603 conn(0x7f5ab0035a30 0x7f5ab0037ee0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5ac806ec00 tx=0x7f5ab4007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5ac0007650 con 0x7f5ac806dfc0 2024-09-18T14:26:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:26:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:26:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 msgr2=0x7f5ac812ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 0x7f5ac812ce50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f7b0 tx=0x7f5ac0009040 comp rx=0 tx=0).stop 2024-09-18T14:26:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 msgr2=0x7f5ac812ce50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:26:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.599+0000 7f5ac57fa700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac806dfc0 0x7f5ac812ce50 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f7b0 tx=0x7f5ac0009040 comp rx=0 tx=0).stop 2024-09-18T14:26:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acf9ff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:26:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acf9ff700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 0x7f5ac812d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acf9ff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80686b0 con 0x7f5ac812d390 2024-09-18T14:26:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 0x7f5ac812d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient: get_auth_request con 0x7f5ac812d390 auth_method 0 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 10 monclient(hunting): handle_auth_done global_id 14733 payload 824 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.611+0000 7f5acd79b700 1 -- 172.21.15.104:0/887604632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5ac8064b30 con 0x7f5ac812d390 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acd79b700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 msgr2=0x7f5ac812d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acd79b700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 0x7f5ac812d790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f7b0 tx=0x7f5ac0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac000de90 con 0x7f5ac812d390 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 10 monclient: dump: 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f5ac0021620 con 0x7f5ac812d390 2024-09-18T14:26:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 msgr2=0x7f5ac812d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 0x7f5ac812d790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f7b0 tx=0x7f5ac0015000 comp rx=0 tx=0).stop 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 msgr2=0x7f5ac812d790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5ac57fa700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812d390 0x7f5ac812d790 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f7b0 tx=0x7f5ac0015000 comp rx=0 tx=0).stop 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:26:04.640 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 30, 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: 1 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 0, 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:26:04.641 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:26:04.642 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:26:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.645 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:26:04.647 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:26:04.648 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:26:04.649 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:26:04.650 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:26:04.651 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:26:04.652 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 >> v2:172.21.15.174:6812/17603 conn(0x7f5ab0035a30 msgr2=0x7f5ab0037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 --2- 172.21.15.104:0/887604632 >> v2:172.21.15.174:6812/17603 conn(0x7f5ab0035a30 0x7f5ab0037ee0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5ac806ec00 tx=0x7f5ab4007e80 comp rx=0 tx=0).stop 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 10 monclient: shutdown 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 msgr2=0x7f5ac806da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 --2- 172.21.15.104:0/887604632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ac812f650 0x7f5ac806da80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8068560 tx=0x7f5ab8017000 comp rx=0 tx=0).stop 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acdf9c700 1 -- 172.21.15.104:0/887604632 reap_dead start 2024-09-18T14:26:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acdf9c700 1 -- 172.21.15.104:0/887604632 reap_dead start 2024-09-18T14:26:04.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 shutdown_connections 2024-09-18T14:26:04.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 >> 172.21.15.104:0/887604632 conn(0x7f5ac80a1290 msgr2=0x7f5ac809fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:26:04.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 shutdown_connections 2024-09-18T14:26:04.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.615+0000 7f5acf9ff700 1 -- 172.21.15.104:0/887604632 wait complete. 2024-09-18T14:26:04.655 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T14:26:04.655 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T14:26:04.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient: get_monmap_and_config 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient: build_initial_monmap 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient: monmap: 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:26:04.764326+0000 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:26:04.764326+0000 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.765 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:26:04.766 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:26:04.766 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:26:04.766 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:26:04.766 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:26:04.766 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 1 Processor -- start 2024-09-18T14:26:04.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 1 -- start start 2024-09-18T14:26:04.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient: init 2024-09-18T14:26:04.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:04.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T14:26:04.768 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2718097c60 0x7f271812fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient(hunting): picked mon.noname-a con 0x7f2718097c60 addr v2:172.21.15.104:3300/0 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f271809adf0 0x7f271809bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient(hunting): picked mon.noname-b con 0x7f271809adf0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f271809b1f0 addr v1:172.21.15.174:6789/0 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271809db70 con 0x7f271809b1f0 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2718099300 con 0x7f2718097c60 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2718098030 con 0x7f271809adf0 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271f8ec700 10 monclient(hunting): authenticate will time out at 1085.753848s 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271ce87700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f271809adf0 msgr2=0x7f271809bc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:04.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271ce87700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f271809adf0 0x7f271809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2718097c60 0x7f271812fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2718097c60 0x7f271812fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41670/0 (socket says 172.21.15.104:41670) 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 -- 172.21.15.104:0/932738901 learned_addr learned my addr 172.21.15.104:0/932738901 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): get_auth_request con 0x7f2718097c60 auth_method 0 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient(hunting): handle_auth_done global_id 14736 payload 274 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 -- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f271809adf0 msgr2=0x7f271809bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 --2- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f271809adf0 0x7f271809bc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 -- 172.21.15.104:0/932738901 >> v1:172.21.15.174:6789/0 conn(0x7f271809b1f0 legacy=0x7f271809c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:26:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 monclient: found mon.noname-a 2024-09-18T14:26:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.763+0000 7f271d688700 1 -- 172.21.15.104:0/932738901 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271809cfa0 con 0x7f2718097c60 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271d688700 1 -- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2718097c60 msgr2=0x7f271812fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271d688700 1 --2- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2718097c60 0x7f271812fb50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f27140099b0 tx=0x7f2714009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99be4c48d240d97 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 1 -- 172.21.15.104:0/932738901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271400bb50 con 0x7f2718097c60 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: dump: 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 1 -- 172.21.15.104:0/932738901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271400bcb0 con 0x7f2718097c60 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 5 monclient: authenticate success, global_id 14736 2024-09-18T14:26:04.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 10 monclient: get_monmap_and_config success 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 1 -- 172.21.15.104:0/932738901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714011650 con 0x7f2718097c60 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 4 set_mon_vals no callback set 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f270ffff700 10 monclient: dump: 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.777 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 10 monclient: shutdown 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 -- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2718097c60 msgr2=0x7f271812fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 --2- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2718097c60 0x7f271812fb50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f27140099b0 tx=0x7f2714009250 comp rx=0 tx=0).stop 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 -- 172.21.15.104:0/932738901 shutdown_connections 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 --2- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f271809adf0 0x7f271809bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 --2- 172.21.15.104:0/932738901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2718097c60 0x7f271812fb50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 -- 172.21.15.104:0/932738901 >> 172.21.15.104:0/932738901 conn(0x7f27180a1290 msgr2=0x7f27180a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 -- 172.21.15.104:0/932738901 shutdown_connections 2024-09-18T14:26:04.778 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 -- 172.21.15.104:0/932738901 wait complete. 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 10 monclient: build_initial_monmap 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 10 monclient: monmap: 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.767+0000 7f271f8ec700 1 Processor -- start 2024-09-18T14:26:04.779 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 1 -- start start 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient: init 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient: _reopen_session rank -1 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2718097c60 0x7f271813f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): picked mon.noname-c con 0x7f2718097c60 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f271809adf0 0x7f271812ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): picked mon.noname-b con 0x7f271809adf0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 0x7f271812f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): picked mon.noname-a con 0x7f271809b1f0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271811c9d0 con 0x7f271809b1f0 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.780 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718098770 con 0x7f271809adf0 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271806e390 con 0x7f2718097c60 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): _renew_subs 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271f8ec700 10 monclient(hunting): authenticate will time out at 1085.760778s 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271d688700 1 -- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2718097c60 msgr2=0x7f271813f540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:3301/0 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271d688700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2718097c60 0x7f271813f540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 0x7f271812f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 0x7f271812f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41676/0 (socket says 172.21.15.104:41676) 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 -- 172.21.15.104:0/2749453959 learned_addr learned my addr 172.21.15.104:0/2749453959 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): get_auth_request con 0x7f271809b1f0 auth_method 0 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.781 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f271809adf0 0x7f271812ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 10 monclient(hunting): get_auth_request con 0x7f271809adf0 auth_method 0 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271ce87700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient(hunting): handle_auth_done global_id 14739 payload 824 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2718097c60 msgr2=0x7f271813f540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:26:04.782 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2718097c60 0x7f271813f540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f271809adf0 msgr2=0x7f271812ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f271809adf0 0x7f271812ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient: _finish_hunting 0 2024-09-18T14:26:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 monclient: found mon.noname-a 2024-09-18T14:26:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:26:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 -- 172.21.15.104:0/2749453959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27140094f0 con 0x7f271809b1f0 2024-09-18T14:26:04.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 msgr2=0x7f271812f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.771+0000 7f271de89700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 0x7f271812f4c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f271000c930 tx=0x7f27100026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710012840 con 0x7f271809b1f0 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: dump: 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: _finish_auth 0 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100129c0 con 0x7f271809b1f0 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 5 monclient: authenticate success, global_id 14739 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 10 monclient: _renew_subs 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271811a570 con 0x7f271809b1f0 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710012d80 con 0x7f271809b1f0 2024-09-18T14:26:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 10 monclient: dump: 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.792 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270f7fe700 4 set_mon_vals no callback set 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 10 monclient: _renew_subs 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271811c150 con 0x7f271809b1f0 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 0x7f2718118e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718068690 con 0x7f2718118aa0 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f271000a5c0 con 0x7f271809b1f0 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 0x7f2718118e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient: get_auth_request con 0x7f2718118aa0 auth_method 0 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 1 --2- 172.21.15.104:0/2749453959 >> v2:172.21.15.174:6812/17603 conn(0x7f26fc035a30 0x7f26fc037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f2710048020 con 0x7f271809b1f0 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 10 monclient(hunting): handle_auth_done global_id 5425 payload 824 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271d688700 1 -- 172.21.15.104:0/2749453959 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2718064b30 con 0x7f2718118aa0 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271ce87700 1 --2- 172.21.15.104:0/2749453959 >> v2:172.21.15.174:6812/17603 conn(0x7f26fc035a30 0x7f26fc037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.775+0000 7f271ce87700 10 monclient: get_auth_request con 0x7f26fc035a30 auth_method 0 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f271d688700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 msgr2=0x7f2718118e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f271d688700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 0x7f2718118e70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2714002000 tx=0x7f2714016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714011ea0 con 0x7f2718118aa0 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 10 monclient: dump: 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.795 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f271ce87700 1 --2- 172.21.15.104:0/2749453959 >> v2:172.21.15.174:6812/17603 conn(0x7f26fc035a30 0x7f26fc037ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f271811a810 tx=0x7f2708009120 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f271401b440 con 0x7f2718118aa0 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 msgr2=0x7f2718118e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 0x7f2718118e70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2714002000 tx=0x7f2714016000 comp rx=0 tx=0).stop 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 msgr2=0x7f2718118e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:26:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.779+0000 7f270d7fa700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718118aa0 0x7f2718118e70 secure :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2714002000 tx=0x7f2714016000 comp rx=0 tx=0).stop 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271f8ec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271f8ec700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 0x7f271806b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271f8ec700 10 monclient(hunting): start opening mon connection 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27180a0500 con 0x7f27181193b0 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 0x7f271806b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient: get_auth_request con 0x7f27181193b0 auth_method 0 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:26:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 10 monclient(hunting): handle_auth_done global_id 5428 payload 824 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.791+0000 7f271d688700 1 -- 172.21.15.104:0/2749453959 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f271809b6d0 con 0x7f27181193b0 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271d688700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 msgr2=0x7f271806b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271d688700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 0x7f271806b000 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f27180687d0 tx=0x7f271401e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714025a90 con 0x7f27181193b0 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 10 monclient: dump: 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:26:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f271401a440 con 0x7f27181193b0 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 msgr2=0x7f271806b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 0x7f271806b000 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f27180687d0 tx=0x7f271401e040 comp rx=0 tx=0).stop 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 30, 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: 1 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.801 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 0, 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.802 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:26:04.803 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:26:04.804 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:26:04.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:26:04.806 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:26:04.807 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.808 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:26:04.809 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:26:04.810 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:26:04.811 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 3 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 7 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:26:04.812 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:26:04.813 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:26:04.813 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:26:04.813 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:26:04.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:26:04.813 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:26:04.813 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:26:04.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 msgr2=0x7f271806b000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:26:04.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f270d7fa700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27181193b0 0x7f271806b000 secure :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f27180687d0 tx=0x7f271401e040 comp rx=0 tx=0).stop 2024-09-18T14:26:04.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 >> v2:172.21.15.174:6812/17603 conn(0x7f26fc035a30 msgr2=0x7f26fc037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 --2- 172.21.15.104:0/2749453959 >> v2:172.21.15.174:6812/17603 conn(0x7f26fc035a30 0x7f26fc037ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f271811a810 tx=0x7f2708009120 comp rx=0 tx=0).stop 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 10 monclient: shutdown 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271de89700 1 -- 172.21.15.104:0/2749453959 reap_dead start 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 msgr2=0x7f271812f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 0x7f271812f4c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f271000c930 tx=0x7f27100026e0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 shutdown_connections 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 --2- 172.21.15.104:0/2749453959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f271809b1f0 0x7f271812f4c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 >> 172.21.15.104:0/2749453959 conn(0x7f27180a1290 msgr2=0x7f271809fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 shutdown_connections 2024-09-18T14:26:04.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:26:04.795+0000 7f271f8ec700 1 -- 172.21.15.104:0/2749453959 wait complete. 2024-09-18T14:26:04.816 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T14:26:04.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T14:26:05.192 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21302]: starting. 2024-09-18T14:26:05.192 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21302]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21301 2024-09-18T14:26:05.193 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21302]: created object 0... 2024-09-18T14:26:05.193 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21302]: created object 25... 2024-09-18T14:26:05.193 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21302]: created object 49... 2024-09-18T14:26:05.193 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21302]: finishing. 2024-09-18T14:26:05.193 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21302]: shutting down. 2024-09-18T14:26:06.001 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21303]: starting. 2024-09-18T14:26:06.001 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21303]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21301 2024-09-18T14:26:06.001 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21303]: shutting down. 2024-09-18T14:26:07.938 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:07.938 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21346]: starting. 2024-09-18T14:26:07.938 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21346]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21301 2024-09-18T14:26:07.938 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21346]: created object 0... 2024-09-18T14:26:07.938 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21346]: created object 25... 2024-09-18T14:26:07.938 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21346]: created object 49... 2024-09-18T14:26:07.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21346]: finishing. 2024-09-18T14:26:07.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21346]: shutting down. 2024-09-18T14:26:07.997 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:07.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21348]: starting. 2024-09-18T14:26:07.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21348]: listing objects. 2024-09-18T14:26:07.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21348]: listed object 0... 2024-09-18T14:26:07.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21348]: listed object 25... 2024-09-18T14:26:07.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21348]: saw 50 objects 2024-09-18T14:26:07.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21348]: shutting down. 2024-09-18T14:26:08.774 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:08.774 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21347]: starting. 2024-09-18T14:26:08.774 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21347]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21301 2024-09-18T14:26:08.774 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21347]: shutting down. 2024-09-18T14:26:08.775 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:08.776 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:08.776 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:08.777 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:10.932 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21399]: starting. 2024-09-18T14:26:10.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21399]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21398 2024-09-18T14:26:10.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21399]: created object 0... 2024-09-18T14:26:10.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21399]: created object 25... 2024-09-18T14:26:10.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21399]: created object 49... 2024-09-18T14:26:10.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21399]: finishing. 2024-09-18T14:26:10.933 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21399]: shutting down. 2024-09-18T14:26:11.802 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21400]: starting. 2024-09-18T14:26:11.802 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21400]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21398 2024-09-18T14:26:11.802 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21400]: shutting down. 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21433]: starting. 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21433]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21398 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21433]: created object 0... 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21433]: created object 25... 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21433]: created object 49... 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21433]: finishing. 2024-09-18T14:26:14.809 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21433]: shutting down. 2024-09-18T14:26:14.869 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:14.869 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21435]: starting. 2024-09-18T14:26:14.869 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21435]: listing objects. 2024-09-18T14:26:14.869 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21435]: listed object 0... 2024-09-18T14:26:14.869 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21435]: listed object 25... 2024-09-18T14:26:14.869 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21435]: saw 50 objects 2024-09-18T14:26:14.869 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21435]: shutting down. 2024-09-18T14:26:15.645 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:15.645 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21434]: starting. 2024-09-18T14:26:15.645 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21434]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21398 2024-09-18T14:26:15.645 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21434]: shutting down. 2024-09-18T14:26:15.646 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:15.646 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:15.646 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:15.648 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:17.832 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21486]: starting. 2024-09-18T14:26:17.832 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21486]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21485 2024-09-18T14:26:17.832 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21486]: created object 0... 2024-09-18T14:26:17.832 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21486]: created object 25... 2024-09-18T14:26:17.832 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21486]: created object 49... 2024-09-18T14:26:17.833 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21486]: finishing. 2024-09-18T14:26:17.833 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21486]: shutting down. 2024-09-18T14:26:18.797 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21487]: starting. 2024-09-18T14:26:18.797 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21487]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21485 2024-09-18T14:26:18.797 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21487]: shutting down. 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21520]: starting. 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21520]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21485 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21520]: created object 0... 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21520]: created object 25... 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21520]: created object 49... 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21520]: finishing. 2024-09-18T14:26:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21520]: shutting down. 2024-09-18T14:26:21.233 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:21.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21522]: starting. 2024-09-18T14:26:21.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21522]: listing objects. 2024-09-18T14:26:21.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21522]: listed object 0... 2024-09-18T14:26:21.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21522]: listed object 25... 2024-09-18T14:26:21.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21522]: saw 50 objects 2024-09-18T14:26:21.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21522]: shutting down. 2024-09-18T14:26:21.816 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:21.816 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21521]: starting. 2024-09-18T14:26:21.816 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21521]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21485 2024-09-18T14:26:21.816 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21521]: shutting down. 2024-09-18T14:26:21.817 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:21.817 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:21.817 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:21.820 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:24.040 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21573]: starting. 2024-09-18T14:26:24.040 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21573]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21572 2024-09-18T14:26:24.040 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21573]: created object 0... 2024-09-18T14:26:24.040 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21573]: created object 25... 2024-09-18T14:26:24.040 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21573]: created object 49... 2024-09-18T14:26:24.040 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21573]: finishing. 2024-09-18T14:26:24.040 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21573]: shutting down. 2024-09-18T14:26:24.840 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21574]: starting. 2024-09-18T14:26:24.840 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21574]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21572 2024-09-18T14:26:24.841 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21574]: shutting down. 2024-09-18T14:26:27.027 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:27.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21607]: starting. 2024-09-18T14:26:27.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21607]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21572 2024-09-18T14:26:27.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21607]: created object 0... 2024-09-18T14:26:27.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21607]: created object 25... 2024-09-18T14:26:27.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21607]: created object 49... 2024-09-18T14:26:27.028 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21607]: finishing. 2024-09-18T14:26:27.029 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21607]: shutting down. 2024-09-18T14:26:27.093 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:27.093 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21609]: starting. 2024-09-18T14:26:27.093 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21609]: listing objects. 2024-09-18T14:26:27.093 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21609]: listed object 0... 2024-09-18T14:26:27.093 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21609]: listed object 25... 2024-09-18T14:26:27.093 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21609]: saw 50 objects 2024-09-18T14:26:27.093 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21609]: shutting down. 2024-09-18T14:26:27.779 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:27.804 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21608]: starting. 2024-09-18T14:26:27.804 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21608]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21572 2024-09-18T14:26:27.804 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21608]: shutting down. 2024-09-18T14:26:27.805 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:27.805 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:27.805 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:27.806 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21660]: starting. 2024-09-18T14:26:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21660]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21659 2024-09-18T14:26:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21660]: created object 0... 2024-09-18T14:26:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21660]: created object 25... 2024-09-18T14:26:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21660]: created object 49... 2024-09-18T14:26:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21660]: finishing. 2024-09-18T14:26:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21660]: shutting down. 2024-09-18T14:26:31.131 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21661]: starting. 2024-09-18T14:26:31.132 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21661]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21659 2024-09-18T14:26:31.132 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21661]: shutting down. 2024-09-18T14:26:33.181 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:33.182 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21694]: starting. 2024-09-18T14:26:33.182 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21694]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21659 2024-09-18T14:26:33.182 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21694]: created object 0... 2024-09-18T14:26:33.182 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21694]: created object 25... 2024-09-18T14:26:33.182 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21694]: created object 49... 2024-09-18T14:26:33.182 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21694]: finishing. 2024-09-18T14:26:33.182 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21694]: shutting down. 2024-09-18T14:26:33.242 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:33.242 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21696]: starting. 2024-09-18T14:26:33.242 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21696]: listing objects. 2024-09-18T14:26:33.242 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21696]: listed object 0... 2024-09-18T14:26:33.242 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21696]: listed object 25... 2024-09-18T14:26:33.242 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21696]: saw 50 objects 2024-09-18T14:26:33.243 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21696]: shutting down. 2024-09-18T14:26:34.015 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:34.015 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21695]: starting. 2024-09-18T14:26:34.015 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21695]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21659 2024-09-18T14:26:34.015 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21695]: shutting down. 2024-09-18T14:26:34.017 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:34.017 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:34.017 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:34.019 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:36.202 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21747]: starting. 2024-09-18T14:26:36.203 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21747]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21746 2024-09-18T14:26:36.203 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21747]: created object 0... 2024-09-18T14:26:36.203 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21747]: created object 25... 2024-09-18T14:26:36.203 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21747]: created object 49... 2024-09-18T14:26:36.203 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21747]: finishing. 2024-09-18T14:26:36.203 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21747]: shutting down. 2024-09-18T14:26:37.041 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21748]: starting. 2024-09-18T14:26:37.041 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21748]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21746 2024-09-18T14:26:37.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21748]: shutting down. 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21781]: starting. 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21781]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21746 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21781]: created object 0... 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21781]: created object 25... 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21781]: created object 49... 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21781]: finishing. 2024-09-18T14:26:39.003 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21781]: shutting down. 2024-09-18T14:26:39.903 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:39.903 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21783]: starting. 2024-09-18T14:26:39.903 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21783]: listing objects. 2024-09-18T14:26:39.903 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21783]: listed object 0... 2024-09-18T14:26:39.903 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21783]: listed object 25... 2024-09-18T14:26:39.903 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21783]: saw 50 objects 2024-09-18T14:26:39.903 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21783]: shutting down. 2024-09-18T14:26:40.800 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:40.800 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21782]: starting. 2024-09-18T14:26:40.800 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21782]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21746 2024-09-18T14:26:40.800 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21782]: shutting down. 2024-09-18T14:26:40.801 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:40.801 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:40.801 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:40.803 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:42.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21834]: starting. 2024-09-18T14:26:42.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21834]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21833 2024-09-18T14:26:42.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21834]: created object 0... 2024-09-18T14:26:42.996 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21834]: created object 25... 2024-09-18T14:26:42.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21834]: created object 49... 2024-09-18T14:26:42.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21834]: finishing. 2024-09-18T14:26:42.997 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21834]: shutting down. 2024-09-18T14:26:43.825 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21835]: starting. 2024-09-18T14:26:43.825 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21835]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21833 2024-09-18T14:26:43.825 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21835]: shutting down. 2024-09-18T14:26:46.057 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:46.057 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21868]: starting. 2024-09-18T14:26:46.057 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21868]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21833 2024-09-18T14:26:46.057 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21868]: created object 0... 2024-09-18T14:26:46.057 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21868]: created object 25... 2024-09-18T14:26:46.058 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21868]: created object 49... 2024-09-18T14:26:46.058 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21868]: finishing. 2024-09-18T14:26:46.058 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21868]: shutting down. 2024-09-18T14:26:46.117 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:46.118 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21870]: starting. 2024-09-18T14:26:46.118 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21870]: listing objects. 2024-09-18T14:26:46.118 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21870]: listed object 0... 2024-09-18T14:26:46.118 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21870]: listed object 25... 2024-09-18T14:26:46.118 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21870]: saw 50 objects 2024-09-18T14:26:46.118 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21870]: shutting down. 2024-09-18T14:26:46.995 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:46.995 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21869]: starting. 2024-09-18T14:26:46.995 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21869]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21833 2024-09-18T14:26:46.995 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21869]: shutting down. 2024-09-18T14:26:47.001 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:47.001 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:47.001 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:47.004 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:48.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21921]: starting. 2024-09-18T14:26:48.986 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21921]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21920 2024-09-18T14:26:48.986 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21921]: created object 0... 2024-09-18T14:26:48.986 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21921]: created object 25... 2024-09-18T14:26:48.986 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21921]: created object 49... 2024-09-18T14:26:48.986 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21921]: finishing. 2024-09-18T14:26:48.986 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[21921]: shutting down. 2024-09-18T14:26:49.790 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21922]: starting. 2024-09-18T14:26:49.791 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21922]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21920 2024-09-18T14:26:49.791 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[21922]: shutting down. 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21956]: starting. 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21956]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-21920 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21956]: created object 0... 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21956]: created object 25... 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21956]: created object 49... 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21956]: finishing. 2024-09-18T14:26:52.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[21956]: shutting down. 2024-09-18T14:26:52.061 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:52.061 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21958]: starting. 2024-09-18T14:26:52.061 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21958]: listing objects. 2024-09-18T14:26:52.061 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21958]: listed object 0... 2024-09-18T14:26:52.061 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21958]: listed object 25... 2024-09-18T14:26:52.061 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21958]: saw 50 objects 2024-09-18T14:26:52.061 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[21958]: shutting down. 2024-09-18T14:26:52.826 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:52.826 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21957]: starting. 2024-09-18T14:26:52.826 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21957]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-21920 2024-09-18T14:26:52.826 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[21957]: shutting down. 2024-09-18T14:26:52.827 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:52.827 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:52.827 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:52.829 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:26:55.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22009]: starting. 2024-09-18T14:26:55.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22009]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22008 2024-09-18T14:26:55.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22009]: created object 0... 2024-09-18T14:26:55.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22009]: created object 25... 2024-09-18T14:26:55.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22009]: created object 49... 2024-09-18T14:26:55.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22009]: finishing. 2024-09-18T14:26:55.002 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22009]: shutting down. 2024-09-18T14:26:55.848 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22010]: starting. 2024-09-18T14:26:55.848 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22010]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22008 2024-09-18T14:26:55.848 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22010]: shutting down. 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22043]: starting. 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22043]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22008 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22043]: created object 0... 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22043]: created object 25... 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22043]: created object 49... 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22043]: finishing. 2024-09-18T14:26:57.963 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22043]: shutting down. 2024-09-18T14:26:58.023 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:58.024 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22045]: starting. 2024-09-18T14:26:58.024 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22045]: listing objects. 2024-09-18T14:26:58.024 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22045]: listed object 0... 2024-09-18T14:26:58.024 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22045]: listed object 25... 2024-09-18T14:26:58.024 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22045]: saw 50 objects 2024-09-18T14:26:58.024 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22045]: shutting down. 2024-09-18T14:26:58.790 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:58.790 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22044]: starting. 2024-09-18T14:26:58.790 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22044]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22008 2024-09-18T14:26:58.791 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22044]: shutting down. 2024-09-18T14:26:58.791 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:58.792 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:26:58.792 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:26:58.793 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:27:01.021 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22097]: starting. 2024-09-18T14:27:01.022 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22097]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22096 2024-09-18T14:27:01.022 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22097]: created object 0... 2024-09-18T14:27:01.022 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22097]: created object 25... 2024-09-18T14:27:01.022 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22097]: created object 49... 2024-09-18T14:27:01.022 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22097]: finishing. 2024-09-18T14:27:01.022 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22097]: shutting down. 2024-09-18T14:27:01.810 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22098]: starting. 2024-09-18T14:27:01.810 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22098]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22096 2024-09-18T14:27:01.810 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22098]: shutting down. 2024-09-18T14:27:03.984 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22131]: starting. 2024-09-18T14:27:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22131]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22096 2024-09-18T14:27:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22131]: created object 0... 2024-09-18T14:27:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22131]: created object 25... 2024-09-18T14:27:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22131]: created object 49... 2024-09-18T14:27:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22131]: finishing. 2024-09-18T14:27:03.985 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22131]: shutting down. 2024-09-18T14:27:04.045 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:04.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22133]: starting. 2024-09-18T14:27:04.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22133]: listing objects. 2024-09-18T14:27:04.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22133]: listed object 0... 2024-09-18T14:27:04.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22133]: listed object 25... 2024-09-18T14:27:04.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22133]: saw 50 objects 2024-09-18T14:27:04.045 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22133]: shutting down. 2024-09-18T14:27:04.841 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:04.841 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22132]: starting. 2024-09-18T14:27:04.841 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22132]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22096 2024-09-18T14:27:04.841 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22132]: shutting down. 2024-09-18T14:27:04.843 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:04.843 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:04.843 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:27:04.846 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:27:07.022 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22184]: starting. 2024-09-18T14:27:07.023 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22184]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22183 2024-09-18T14:27:07.023 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22184]: created object 0... 2024-09-18T14:27:07.023 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22184]: created object 25... 2024-09-18T14:27:07.023 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22184]: created object 49... 2024-09-18T14:27:07.023 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22184]: finishing. 2024-09-18T14:27:07.023 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22184]: shutting down. 2024-09-18T14:27:07.796 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22185]: starting. 2024-09-18T14:27:07.797 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22185]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22183 2024-09-18T14:27:07.797 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22185]: shutting down. 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22218]: starting. 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22218]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22183 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22218]: created object 0... 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22218]: created object 25... 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22218]: created object 49... 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22218]: finishing. 2024-09-18T14:27:09.975 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22218]: shutting down. 2024-09-18T14:27:10.033 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:10.033 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22220]: starting. 2024-09-18T14:27:10.033 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22220]: listing objects. 2024-09-18T14:27:10.033 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22220]: listed object 0... 2024-09-18T14:27:10.033 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22220]: listed object 25... 2024-09-18T14:27:10.033 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22220]: saw 50 objects 2024-09-18T14:27:10.033 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22220]: shutting down. 2024-09-18T14:27:10.813 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:10.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22219]: starting. 2024-09-18T14:27:10.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22219]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22183 2024-09-18T14:27:10.814 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22219]: shutting down. 2024-09-18T14:27:10.814 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:10.814 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:10.814 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:27:10.816 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:27:13.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22271]: starting. 2024-09-18T14:27:13.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22271]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22270 2024-09-18T14:27:13.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22271]: created object 0... 2024-09-18T14:27:13.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22271]: created object 25... 2024-09-18T14:27:13.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22271]: created object 49... 2024-09-18T14:27:13.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22271]: finishing. 2024-09-18T14:27:13.027 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22271]: shutting down. 2024-09-18T14:27:13.856 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22272]: starting. 2024-09-18T14:27:13.856 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22272]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22270 2024-09-18T14:27:13.856 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22272]: shutting down. 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22305]: starting. 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22305]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22270 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22305]: created object 0... 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22305]: created object 25... 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22305]: created object 49... 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22305]: finishing. 2024-09-18T14:27:16.042 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22305]: shutting down. 2024-09-18T14:27:16.102 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:16.102 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22307]: starting. 2024-09-18T14:27:16.102 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22307]: listing objects. 2024-09-18T14:27:16.102 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22307]: listed object 0... 2024-09-18T14:27:16.102 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22307]: listed object 25... 2024-09-18T14:27:16.102 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22307]: saw 50 objects 2024-09-18T14:27:16.102 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22307]: shutting down. 2024-09-18T14:27:16.879 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:16.879 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22306]: starting. 2024-09-18T14:27:16.880 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22306]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22270 2024-09-18T14:27:16.880 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22306]: shutting down. 2024-09-18T14:27:16.881 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:16.881 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:16.881 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:27:16.882 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:27:18.959 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22358]: starting. 2024-09-18T14:27:18.959 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22358]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22357 2024-09-18T14:27:18.959 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22358]: created object 0... 2024-09-18T14:27:18.959 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22358]: created object 25... 2024-09-18T14:27:18.959 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22358]: created object 49... 2024-09-18T14:27:18.959 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22358]: finishing. 2024-09-18T14:27:18.959 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22358]: shutting down. 2024-09-18T14:27:19.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22359]: starting. 2024-09-18T14:27:19.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22359]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22357 2024-09-18T14:27:19.806 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22359]: shutting down. 2024-09-18T14:27:22.006 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:22.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22392]: starting. 2024-09-18T14:27:22.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22392]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22357 2024-09-18T14:27:22.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22392]: created object 0... 2024-09-18T14:27:22.006 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22392]: created object 25... 2024-09-18T14:27:22.007 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22392]: created object 49... 2024-09-18T14:27:22.007 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22392]: finishing. 2024-09-18T14:27:22.007 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22392]: shutting down. 2024-09-18T14:27:22.068 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:22.068 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22394]: starting. 2024-09-18T14:27:22.068 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22394]: listing objects. 2024-09-18T14:27:22.069 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22394]: listed object 0... 2024-09-18T14:27:22.069 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22394]: listed object 25... 2024-09-18T14:27:22.069 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22394]: saw 50 objects 2024-09-18T14:27:22.069 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22394]: shutting down. 2024-09-18T14:27:22.843 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:22.843 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22393]: starting. 2024-09-18T14:27:22.843 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22393]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22357 2024-09-18T14:27:22.843 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22393]: shutting down. 2024-09-18T14:27:22.844 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:22.844 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:22.845 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:27:22.846 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:27:25.033 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22445]: starting. 2024-09-18T14:27:25.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22445]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22444 2024-09-18T14:27:25.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22445]: created object 0... 2024-09-18T14:27:25.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22445]: created object 25... 2024-09-18T14:27:25.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22445]: created object 49... 2024-09-18T14:27:25.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22445]: finishing. 2024-09-18T14:27:25.034 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22445]: shutting down. 2024-09-18T14:27:25.865 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22446]: starting. 2024-09-18T14:27:25.865 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22446]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22444 2024-09-18T14:27:25.865 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22446]: shutting down. 2024-09-18T14:27:27.939 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:27.939 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22479]: starting. 2024-09-18T14:27:27.940 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22479]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22444 2024-09-18T14:27:27.940 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22479]: created object 0... 2024-09-18T14:27:27.940 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22479]: created object 25... 2024-09-18T14:27:27.940 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22479]: created object 49... 2024-09-18T14:27:27.940 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22479]: finishing. 2024-09-18T14:27:27.940 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22479]: shutting down. 2024-09-18T14:27:27.999 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:27.999 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22481]: starting. 2024-09-18T14:27:27.999 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22481]: listing objects. 2024-09-18T14:27:27.999 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22481]: listed object 0... 2024-09-18T14:27:28.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22481]: listed object 25... 2024-09-18T14:27:28.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22481]: saw 50 objects 2024-09-18T14:27:28.000 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22481]: shutting down. 2024-09-18T14:27:28.805 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:28.805 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22480]: starting. 2024-09-18T14:27:28.805 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22480]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22444 2024-09-18T14:27:28.805 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22480]: shutting down. 2024-09-18T14:27:28.806 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:28.806 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:28.806 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:27:28.808 DEBUG:teuthology.orchestra.run.smithi174:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-18T14:27:30.994 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22532]: starting. 2024-09-18T14:27:30.994 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22532]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22531 2024-09-18T14:27:30.994 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22532]: created object 0... 2024-09-18T14:27:30.994 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22532]: created object 25... 2024-09-18T14:27:30.994 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22532]: created object 49... 2024-09-18T14:27:30.994 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22532]: finishing. 2024-09-18T14:27:30.994 INFO:teuthology.orchestra.run.smithi174.stdout:process_1_[22532]: shutting down. 2024-09-18T14:27:31.836 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22533]: starting. 2024-09-18T14:27:31.837 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22533]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22531 2024-09-18T14:27:31.837 INFO:teuthology.orchestra.run.smithi174.stdout:process_2_[22533]: shutting down. 2024-09-18T14:27:34.173 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:34.173 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22566]: starting. 2024-09-18T14:27:34.173 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22566]: creating pool ceph_test_rados_delete_pools_parallel.smithi174-22531 2024-09-18T14:27:34.173 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22566]: created object 0... 2024-09-18T14:27:34.174 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22566]: created object 25... 2024-09-18T14:27:34.174 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22566]: created object 49... 2024-09-18T14:27:34.174 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22566]: finishing. 2024-09-18T14:27:34.174 INFO:teuthology.orchestra.run.smithi174.stdout:process_3_[22566]: shutting down. 2024-09-18T14:27:34.234 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:34.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22568]: starting. 2024-09-18T14:27:34.234 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22568]: listing objects. 2024-09-18T14:27:34.235 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22568]: listed object 0... 2024-09-18T14:27:34.235 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22568]: listed object 25... 2024-09-18T14:27:34.235 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22568]: saw 50 objects 2024-09-18T14:27:34.235 INFO:teuthology.orchestra.run.smithi174.stdout:process_5_[22568]: shutting down. 2024-09-18T14:27:34.816 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T14:27:34.816 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T14:27:34.816 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T14:27:34.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T14:27:34.819 INFO:tasks.ceph.mon.c:Started 2024-09-18T14:27:34.820 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T14:27:34.820 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:27:34.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient: get_monmap_and_config 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient: build_initial_monmap 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient: monmap: 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:34.920321+0000 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:34.920321+0000 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:34.922 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:34.923 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:34.923 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:34.923 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:34.923 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:34.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 1 Processor -- start 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 1 -- start start 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient: init 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c0a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient(hunting): picked mon.noname-b con 0x7fbc8c09fe10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbc8c09c900 0x7fbc8c0a0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient(hunting): picked mon.noname-c con 0x7fbc8c09c900 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc8c09c090 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c09b990 con 0x7fbc8c09c090 2024-09-18T14:27:34.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c0a1320 con 0x7fbc8c09c900 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0a0c60 con 0x7fbc8c09fe10 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc90d9f700 10 monclient(hunting): authenticate will time out at 1175.910307s 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c0a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c0a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:50440/0 (socket says 172.21.15.104:50440) 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/3444916632 learned_addr learned my addr 172.21.15.104:0/3444916632 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 10 monclient(hunting): get_auth_request con 0x7fbc8c09fe10 auth_method 0 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:34.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/3444916632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 msgr2=0x7fbc8c0a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/3444916632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 msgr2=0x7fbc8c0a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 1 --2- 172.21.15.104:0/3444916632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c0a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8b7fe700 1 --2- 172.21.15.104:0/3444916632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c0a01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 1 --2- 172.21.15.104:0/3444916632 >> v2:172.21.15.174:3300/0 conn(0x7fbc8c09c900 0x7fbc8c0a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): get_auth_request con 0x7fbc8c09c900 auth_method 0 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:34.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient(hunting): handle_auth_done global_id 5776 payload 274 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 1 -- 172.21.15.104:0/3444916632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 msgr2=0x7fbc8c0a01e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 1 --2- 172.21.15.104:0/3444916632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c0a01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 1 -- 172.21.15.104:0/3444916632 >> v1:172.21.15.104:6789/0 conn(0x7fbc8c09c090 legacy=0x7fbc8c131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient: _finish_hunting 0 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 1 monclient: found mon.noname-c 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:27:34.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.917+0000 7fbc8affd700 1 -- 172.21.15.104:0/3444916632 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c09b6b0 con 0x7fbc8c09c900 2024-09-18T14:27:34.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8affd700 1 -- 172.21.15.104:0/3444916632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 msgr2=0x7fbc8c0a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:34.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8affd700 1 --2- 172.21.15.104:0/3444916632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 0x7fbc8c0a0720 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbc840098e0 tx=0x7fbc840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c770cc4e5f447dc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:34.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 1 -- 172.21.15.104:0/3444916632 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84009e80 con 0x7fbc8c09c900 2024-09-18T14:27:34.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:34.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:34.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: dump: 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:34.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 1 -- 172.21.15.104:0/3444916632 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8400ba70 con 0x7fbc8c09c900 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 1 -- 172.21.15.104:0/3444916632 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8400bdf0 con 0x7fbc8c09c900 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 5 monclient: authenticate success, global_id 5776 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 4 set_mon_vals no callback set 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc8a7fc700 10 monclient: dump: 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.937 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 10 monclient: shutdown 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 -- 172.21.15.104:0/3444916632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 msgr2=0x7fbc8c0a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/3444916632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 0x7fbc8c0a0720 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbc840098e0 tx=0x7fbc840091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 -- 172.21.15.104:0/3444916632 shutdown_connections 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/3444916632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 0x7fbc8c0a0720 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:34.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/3444916632 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c0a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 -- 172.21.15.104:0/3444916632 >> 172.21.15.104:0/3444916632 conn(0x7fbc8c0a6080 msgr2=0x7fbc8c0676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 -- 172.21.15.104:0/3444916632 shutdown_connections 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.921+0000 7fbc90d9f700 1 -- 172.21.15.104:0/3444916632 wait complete. 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient: build_initial_monmap 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient: monmap: 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:34.939 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 Processor -- start 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 -- start start 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient: init 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 0x7fbc8c141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): picked mon.noname-a con 0x7fbc8c09c090 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:34.940 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 0x7fbc8c141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): picked mon.noname-b con 0x7fbc8c09c900 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c11e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): picked mon.noname-c con 0x7fbc8c09fe10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c121d70 con 0x7fbc8c09c090 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c120cb0 con 0x7fbc8c09c900 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c09ae40 con 0x7fbc8c09fe10 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc90d9f700 10 monclient(hunting): authenticate will time out at 1175.917995s 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 0x7fbc8c141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:34.941 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 0x7fbc8c141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39674/0 (socket says 172.21.15.104:39674) 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/742449892 learned_addr learned my addr 172.21.15.104:0/742449892 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): get_auth_request con 0x7fbc8c09c090 auth_method 0 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c11e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 10 monclient(hunting): get_auth_request con 0x7fbc8c09fe10 auth_method 0 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:34.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 0x7fbc8c141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 1 -- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 msgr2=0x7fbc8c11e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 1 -- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 msgr2=0x7fbc8c11e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c11e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 10 monclient(hunting): get_auth_request con 0x7fbc8c09c900 auth_method 0 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8bfff700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c11e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:34.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient(hunting): handle_auth_done global_id 15117 payload 824 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 msgr2=0x7fbc8c11e7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c11e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/742449892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 msgr2=0x7fbc8c141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 0x7fbc8c141b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 monclient: found mon.noname-a 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.925+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/742449892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc84009420 con 0x7fbc8c09c090 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc8b7fe700 1 -- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 msgr2=0x7fbc8c141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc8b7fe700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 0x7fbc8c141600 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c007e80 tx=0x7fbc7c0092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 1 -- 172.21.15.104:0/742449892 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00dd70 con 0x7fbc8c09c090 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: dump: 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:34.945 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: _finish_auth 0 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 1 -- 172.21.15.104:0/742449892 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c00f040 con 0x7fbc8c09c090 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 5 monclient: authenticate success, global_id 15117 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 10 monclient: _renew_subs 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c09e660 con 0x7fbc8c09c090 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 1 -- 172.21.15.104:0/742449892 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c013640 con 0x7fbc8c09c090 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 10 monclient: dump: 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:34.946 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc89ffb700 4 set_mon_vals no callback set 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 10 monclient: _renew_subs 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:34.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c1202b0 con 0x7fbc8c09c090 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c0633b0 con 0x7fbc8c09c090 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 1 -- 172.21.15.104:0/742449892 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fbc7c003020 con 0x7fbc8c09c090 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.929+0000 7fbc837fe700 1 --2- 172.21.15.104:0/742449892 >> v2:172.21.15.174:6812/17603 conn(0x7fbc6c035a80 0x7fbc6c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.933+0000 7fbc837fe700 1 -- 172.21.15.104:0/742449892 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(325..325 src has 124..325) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fbc7c047c10 con 0x7fbc8c09c090 2024-09-18T14:27:34.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.933+0000 7fbc8affd700 1 --2- 172.21.15.104:0/742449892 >> v2:172.21.15.174:6812/17603 conn(0x7fbc6c035a80 0x7fbc6c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:34.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.933+0000 7fbc8affd700 10 monclient: get_auth_request con 0x7fbc6c035a80 auth_method 0 2024-09-18T14:27:34.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.933+0000 7fbc8affd700 1 --2- 172.21.15.104:0/742449892 >> v2:172.21.15.174:6812/17603 conn(0x7fbc6c035a80 0x7fbc6c037f30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbc840093f0 tx=0x7fbc84016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:34.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.941+0000 7fbc837fe700 1 -- 172.21.15.104:0/742449892 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbc7c002880 con 0x7fbc8c09c090 2024-09-18T14:27:34.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.941+0000 7fbc837fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:34.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:34.941+0000 7fbc837fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:35.019 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:35.020 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22567]: starting. 2024-09-18T14:27:35.020 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22567]: deleting pool ceph_test_rados_delete_pools_parallel.smithi174-22531 2024-09-18T14:27:35.020 INFO:teuthology.orchestra.run.smithi174.stdout:process_4_[22567]: shutting down. 2024-09-18T14:27:35.020 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:35.020 INFO:teuthology.orchestra.run.smithi174.stdout:******************************* 2024-09-18T14:27:35.020 INFO:teuthology.orchestra.run.smithi174.stdout:******* SUCCESS ********** 2024-09-18T14:27:35.021 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-18T14:27:35.032 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-18T14:27:35.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:27:35.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbc8c121f90 con 0x7fbc8c09c090 2024-09-18T14:27:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc837fe700 1 -- 172.21.15.104:0/742449892 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fbc7c017070 con 0x7fbc8c09c090 2024-09-18T14:27:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc837fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc837fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:35.182 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:27:35.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 >> v2:172.21.15.174:6812/17603 conn(0x7fbc6c035a80 msgr2=0x7fbc6c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/742449892 >> v2:172.21.15.174:6812/17603 conn(0x7fbc6c035a80 0x7fbc6c037f30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbc840093f0 tx=0x7fbc84016000 comp rx=0 tx=0).stop 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 10 monclient: shutdown 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 msgr2=0x7fbc8c141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 0x7fbc8c141600 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c007e80 tx=0x7fbc7c0092f0 comp rx=0 tx=0).stop 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 shutdown_connections 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/742449892 >> v2:172.21.15.174:6812/17603 conn(0x7fbc6c035a80 0x7fbc6c037f30 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbc8c09fe10 0x7fbc8c11e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc8c09c900 0x7fbc8c141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 --2- 172.21.15.104:0/742449892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc8c09c090 0x7fbc8c141600 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 >> 172.21.15.104:0/742449892 conn(0x7fbc8c0a6080 msgr2=0x7fbc8c09cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 shutdown_connections 2024-09-18T14:27:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:35.177+0000 7fbc90d9f700 1 -- 172.21.15.104:0/742449892 wait complete. 2024-09-18T14:27:38.197 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:27:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient: get_monmap_and_config 2024-09-18T14:27:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient: build_initial_monmap 2024-09-18T14:27:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient: monmap: 2024-09-18T14:27:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:38.305712+0000 2024-09-18T14:27:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:38.305712+0000 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:38.308 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 Processor -- start 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 -- start start 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient: init 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f389009fe10 0x7f38900a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): picked mon.noname-c con 0x7f389009fe10 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 0x7f38900a0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:38.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): picked mon.noname-b con 0x7f389009c900 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f389009c090 0x7f3890131be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): picked mon.noname-a con 0x7f389009c090 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38900a1320 con 0x7f389009c090 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389009b990 con 0x7f389009fe10 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900a0c60 con 0x7f389009c900 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3896cf2700 10 monclient(hunting): authenticate will time out at 1179.295262s 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 0x7f38900a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:38.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 0x7f38900a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:50456/0 (socket says 172.21.15.104:50456) 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 1 -- 172.21.15.104:0/1239845629 learned_addr learned my addr 172.21.15.104:0/1239845629 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f389009c090 0x7f3890131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 10 monclient(hunting): get_auth_request con 0x7f389009c900 auth_method 0 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 10 monclient(hunting): get_auth_request con 0x7f389009c090 auth_method 0 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3895cf0700 1 --2- 172.21.15.104:0/1239845629 >> v2:172.21.15.174:3300/0 conn(0x7f389009fe10 0x7f38900a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:38.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3895cf0700 10 monclient(hunting): get_auth_request con 0x7f389009fe10 auth_method 0 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3895cf0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3895cf0700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3895cf0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 1 -- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 msgr2=0x7f38900a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 1 -- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 msgr2=0x7f38900a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 1 --2- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 0x7f38900a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38964f1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f38954ef700 1 --2- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 0x7f38900a0720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3895cf0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:38.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.301+0000 7f3895cf0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:38.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3895cf0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:38.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 10 monclient(hunting): handle_auth_done global_id 15120 payload 274 2024-09-18T14:27:38.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 -- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 msgr2=0x7f38900a0720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:27:38.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 --2- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 0x7f38900a0720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 -- 172.21.15.104:0/1239845629 >> v2:172.21.15.174:3300/0 conn(0x7f389009fe10 msgr2=0x7f38900a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 --2- 172.21.15.104:0/1239845629 >> v2:172.21.15.174:3300/0 conn(0x7f389009fe10 0x7f38900a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 10 monclient: _finish_hunting 0 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 monclient: found mon.noname-a 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 -- 172.21.15.104:0/1239845629 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389009b6b0 con 0x7f389009c090 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 -- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c090 msgr2=0x7f3890131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f38964f1700 1 --2- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c090 0x7f3890131be0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f389009a470 tx=0x7f388c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d059e5d2ba9a8c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 1 -- 172.21.15.104:0/1239845629 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c011840 con 0x7f389009c090 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:38.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: dump: 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:38.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: _finish_auth 0 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 5 monclient: authenticate success, global_id 15120 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 1 -- 172.21.15.104:0/1239845629 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c0119a0 con 0x7f389009c090 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 4 set_mon_vals no callback set 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 1 -- 172.21.15.104:0/1239845629 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c011d60 con 0x7f389009c090 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3894cee700 10 monclient: dump: 2024-09-18T14:27:38.321 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 10 monclient: shutdown 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 -- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c090 msgr2=0x7f3890131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:38.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c090 0x7f3890131be0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f389009a470 tx=0x7f388c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 -- 172.21.15.104:0/1239845629 shutdown_connections 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c090 0x7f3890131be0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1239845629 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c900 0x7f38900a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1239845629 >> v2:172.21.15.174:3300/0 conn(0x7f389009fe10 0x7f38900a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 -- 172.21.15.104:0/1239845629 >> 172.21.15.104:0/1239845629 conn(0x7f38900a6080 msgr2=0x7f38900676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 -- 172.21.15.104:0/1239845629 shutdown_connections 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.305+0000 7f3896cf2700 1 -- 172.21.15.104:0/1239845629 wait complete. 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient: build_initial_monmap 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient: monmap: 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:38.323 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:38.324 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:38.324 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:38.324 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:38.324 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:38.324 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:38.324 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 Processor -- start 2024-09-18T14:27:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 -- start start 2024-09-18T14:27:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient: init 2024-09-18T14:27:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:38.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 0x7f389011e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): picked mon.noname-c con 0x7f389009c090 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 0x7f389011ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): picked mon.noname-a con 0x7f389009c900 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f389009fe10 0x7f389011f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): picked mon.noname-b con 0x7f389009fe10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389009ae40 con 0x7f389009c900 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3890122cf0 con 0x7f389009fe10 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900a3220 con 0x7f389009c090 2024-09-18T14:27:38.332 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3896cf2700 10 monclient(hunting): authenticate will time out at 1179.303431s 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 0x7f389011e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 0x7f389011ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 0x7f389011e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:50468/0 (socket says 172.21.15.104:50468) 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 1 -- 172.21.15.104:0/1419465420 learned_addr learned my addr 172.21.15.104:0/1419465420 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 0x7f389011ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39736/0 (socket says 172.21.15.104:39736) 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 10 monclient(hunting): get_auth_request con 0x7f389009c900 auth_method 0 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 10 monclient(hunting): get_auth_request con 0x7f389009c090 auth_method 0 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:38.333 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38954ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 1 -- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 msgr2=0x7f389011e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f38964f1700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f389009fe10 0x7f389011f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 1 -- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 msgr2=0x7f389011e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.309+0000 7f3895cf0700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 0x7f389011e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38964f1700 10 monclient(hunting): get_auth_request con 0x7f389009fe10 auth_method 0 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38964f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38964f1700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38964f1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:38.334 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 10 monclient(hunting): handle_auth_done global_id 15123 payload 824 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3895cf0700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 0x7f389011e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 -- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 msgr2=0x7f389011e6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 0x7f389011e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 -- 172.21.15.104:0/1419465420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f389009fe10 msgr2=0x7f389011f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f389009fe10 0x7f389011f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 10 monclient: _finish_hunting 0 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 monclient: found mon.noname-a 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 -- 172.21.15.104:0/1419465420 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c009420 con 0x7f389009c900 2024-09-18T14:27:38.335 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 -- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 msgr2=0x7f389011ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38954ef700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 0x7f389011ebe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3880009060 tx=0x7f3880008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 1 -- 172.21.15.104:0/1419465420 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880013620 con 0x7f389009c900 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: dump: 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:38.336 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3896cf2700 5 monclient: authenticate success, global_id 15123 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 1 -- 172.21.15.104:0/1419465420 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38800137a0 con 0x7f389009c900 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3896cf2700 10 monclient: _renew_subs 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 1 -- 172.21.15.104:0/1419465420 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880013b60 con 0x7f389009c900 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3896cf2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f38877fe700 4 set_mon_vals no callback set 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3890122000 con 0x7f389009c900 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:38.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3885ffb700 10 monclient: dump: 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3896cf2700 10 monclient: _renew_subs 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3896cf2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.313+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3890120860 con 0x7f389009c900 2024-09-18T14:27:38.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3896cf2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3896cf2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3885ffb700 1 -- 172.21.15.104:0/1419465420 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f388001b400 con 0x7f389009c900 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3896cf2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38900633b0 con 0x7f389009c900 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3885ffb700 1 --2- 172.21.15.104:0/1419465420 >> v2:172.21.15.174:6812/17603 conn(0x7f3874035a80 0x7f3874037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3885ffb700 1 -- 172.21.15.104:0/1419465420 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3880047ed0 con 0x7f389009c900 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3895cf0700 1 --2- 172.21.15.104:0/1419465420 >> v2:172.21.15.174:6812/17603 conn(0x7f3874035a80 0x7f3874037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.317+0000 7f3895cf0700 10 monclient: get_auth_request con 0x7f3874035a80 auth_method 0 2024-09-18T14:27:38.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.321+0000 7f3895cf0700 1 --2- 172.21.15.104:0/1419465420 >> v2:172.21.15.174:6812/17603 conn(0x7f3874035a80 0x7f3874037f30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f38880099f0 tx=0x7f388800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.325+0000 7f3885ffb700 1 -- 172.21.15.104:0/1419465420 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3880022900 con 0x7f389009c900 2024-09-18T14:27:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.325+0000 7f3885ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:38.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.325+0000 7f3885ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:38.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.545+0000 7f3896cf2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:27:38.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.545+0000 7f3896cf2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:38.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.545+0000 7f3896cf2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:38.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.545+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f389011ff10 con 0x7f389009c900 2024-09-18T14:27:38.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.545+0000 7f3885ffb700 1 -- 172.21.15.104:0/1419465420 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f3880010320 con 0x7f389009c900 2024-09-18T14:27:38.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.545+0000 7f3885ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:38.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.545+0000 7f3885ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:38.553 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:27:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 >> v2:172.21.15.174:6812/17603 conn(0x7f3874035a80 msgr2=0x7f3874037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1419465420 >> v2:172.21.15.174:6812/17603 conn(0x7f3874035a80 0x7f3874037f30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f38880099f0 tx=0x7f388800b040 comp rx=0 tx=0).stop 2024-09-18T14:27:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 10 monclient: shutdown 2024-09-18T14:27:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:38.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 msgr2=0x7f389011ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 0x7f389011ebe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3880009060 tx=0x7f3880008040 comp rx=0 tx=0).stop 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 shutdown_connections 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1419465420 >> v2:172.21.15.174:6812/17603 conn(0x7f3874035a80 0x7f3874037f30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f389009fe10 0x7f389011f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f389009c900 0x7f389011ebe0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 --2- 172.21.15.104:0/1419465420 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f389009c090 0x7f389011e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 >> 172.21.15.104:0/1419465420 conn(0x7f38900a6080 msgr2=0x7f389009cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 shutdown_connections 2024-09-18T14:27:38.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:38.549+0000 7f3896cf2700 1 -- 172.21.15.104:0/1419465420 wait complete. 2024-09-18T14:27:41.567 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:27:41.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.669+0000 7fbe6b097700 10 monclient: get_monmap_and_config 2024-09-18T14:27:41.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.669+0000 7fbe6b097700 10 monclient: build_initial_monmap 2024-09-18T14:27:41.676 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.669+0000 7fbe6b097700 10 monclient: monmap: 2024-09-18T14:27:41.676 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:41.676 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:41.676 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:41.674778+0000 2024-09-18T14:27:41.676 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:41.674778+0000 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:41.677 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.669+0000 7fbe6b097700 1 Processor -- start 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.669+0000 7fbe6b097700 1 -- start start 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.669+0000 7fbe6b097700 10 monclient: init 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe64132170 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe64096a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient(hunting): picked mon.noname-b con 0x7fbe6409ed30 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe6406a8e0 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6409fcf0 con 0x7fbe6406a8e0 2024-09-18T14:27:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6409e5c0 con 0x7fbe64132170 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6406be70 con 0x7fbe6409ed30 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe6b097700 10 monclient(hunting): authenticate will time out at 1182.664629s 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe64096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe64096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:57900/0 (socket says 172.21.15.104:57900) 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe69634700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbe6406a8e0 0x7fbe64139430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:49388/0 (socket says 172.21.15.104:49388) 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 1 -- 172.21.15.104:0/3319007141 learned_addr learned my addr 172.21.15.104:0/3319007141 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 10 monclient(hunting): get_auth_request con 0x7fbe6409ed30 auth_method 0 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 1 -- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 msgr2=0x7fbe64096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 1 -- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 msgr2=0x7fbe64096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:27:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe64096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:41.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe63fff700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe64096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:41.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe69634700 1 -- 172.21.15.104:0/3319007141 >> v1:172.21.15.104:6789/0 conn(0x7fbe6406a8e0 legacy=0x7fbe64139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:41.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe68e33700 1 -- 172.21.15.104:0/3319007141 >> v1:172.21.15.174:6789/0 conn(0x7fbe64132170 legacy=0x7fbe64132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:41.685 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020995412 0 0) 0x7fbe6409fcf0 con 0x7fbe6406a8e0 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 10 monclient(hunting): my global_id is 15126 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe50003570 con 0x7fbe6406a8e0 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965788120 0 0) 0x7fbe6409e5c0 con 0x7fbe64132170 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 10 monclient(hunting): my global_id is 5788 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe6409fcf0 con 0x7fbe64132170 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490166457 0 0) 0x7fbe50003570 con 0x7fbe6406a8e0 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 msgr2=0x7fbe64096a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe64096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.673+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 >> v1:172.21.15.174:6789/0 conn(0x7fbe64132170 legacy=0x7fbe64132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: _finish_hunting 0 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 monclient: found mon.noname-a-legacy 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.104:6789/0 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6409f730 con 0x7fbe6406a8e0 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: _finish_auth 0 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7fbe5c0017a0 con 0x7fbe6406a8e0 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe6b097700 5 monclient: authenticate success, global_id 15126 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe6b097700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: dump: 2024-09-18T14:27:41.687 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 >> v1:172.21.15.104:6789/0 conn(0x7fbe6406a8e0 legacy=0x7fbe64139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:27:41.688 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe50003d90 0x7fbe50004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): picked mon.b con 0x7fbe50003d90 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe50004cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): picked mon.a con 0x7fbe50004860 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe5000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): picked mon.c con 0x7fbe6409ed30 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6406be70 con 0x7fbe50004860 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe500056d0 con 0x7fbe50003d90 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5000c020 con 0x7fbe6409ed30 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:27:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe6b097700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe50004cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe5000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): get_auth_request con 0x7fbe50004860 auth_method 0 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 10 monclient(hunting): get_auth_request con 0x7fbe6409ed30 auth_method 0 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe68e33700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe50003d90 0x7fbe50004180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.690 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe69634700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe68e33700 10 monclient(hunting): get_auth_request con 0x7fbe50003d90 auth_method 0 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe68e33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe68e33700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe68e33700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient(hunting): handle_auth_done global_id 15126 payload 293 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 -- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 msgr2=0x7fbe5000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe6409ed30 0x7fbe5000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 -- 172.21.15.104:0/3319007141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe50003d90 msgr2=0x7fbe50004180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe50003d90 0x7fbe50004180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient: _finish_hunting 0 2024-09-18T14:27:41.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 monclient: found mon.a 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 -- 172.21.15.104:0/3319007141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe50003570 con 0x7fbe50004860 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 10 monclient: _finish_auth 0 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 -- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 msgr2=0x7fbe50004cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe63fff700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe50004cb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009060 tx=0x7fbe4c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c421a849ed564c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c00c070 con 0x7fbe50004860 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: dump: 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:41.692 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe6b097700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4c00e5e0 con 0x7fbe50004860 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe6b097700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe6b097700 4 set_mon_vals no callback set 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 1 -- 172.21.15.104:0/3319007141 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c00e9a0 con 0x7fbe50004860 2024-09-18T14:27:41.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.677+0000 7fbe637fe700 10 monclient: dump: 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient: shutdown 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:41.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 msgr2=0x7fbe50004cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:41.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 --2- 172.21.15.104:0/3319007141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe50004cb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009060 tx=0x7fbe4c008040 comp rx=0 tx=0).stop 2024-09-18T14:27:41.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 -- 172.21.15.104:0/3319007141 reap_dead start 2024-09-18T14:27:41.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- 172.21.15.104:0/3319007141 shutdown_connections 2024-09-18T14:27:41.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- 172.21.15.104:0/3319007141 >> 172.21.15.104:0/3319007141 conn(0x7fbe640a3420 msgr2=0x7fbe640a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:41.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- 172.21.15.104:0/3319007141 shutdown_connections 2024-09-18T14:27:41.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- 172.21.15.104:0/3319007141 wait complete. 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient: build_initial_monmap 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient: monmap: 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:41.700 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 Processor -- start 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- start start 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient: init 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe64141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): picked mon.noname-a con 0x7fbe50004860 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe64131180 0x7fbe64097120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): picked mon.noname-b con 0x7fbe64131180 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 0x7fbe64097660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): picked mon.noname-c con 0x7fbe64131550 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:41.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6409f150 con 0x7fbe50004860 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6406d720 con 0x7fbe64131180 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6411e330 con 0x7fbe64131550 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe6b097700 10 monclient(hunting): authenticate will time out at 1182.674525s 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 0x7fbe64097660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe64141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 0x7fbe64097660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:57920/0 (socket says 172.21.15.104:57920) 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 -- 172.21.15.104:0/1736973357 learned_addr learned my addr 172.21.15.104:0/1736973357 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:41.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): get_auth_request con 0x7fbe50004860 auth_method 0 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 10 monclient(hunting): get_auth_request con 0x7fbe64131550 auth_method 0 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe64131180 0x7fbe64097120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 10 monclient(hunting): get_auth_request con 0x7fbe64131180 auth_method 0 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:41.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 -- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 msgr2=0x7fbe64097660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 -- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 msgr2=0x7fbe64097660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 0x7fbe64097660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe69634700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 0x7fbe64097660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient(hunting): handle_auth_done global_id 15129 payload 824 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 1 -- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 msgr2=0x7fbe64097660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 0x7fbe64097660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 1 -- 172.21.15.104:0/1736973357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe64131180 msgr2=0x7fbe64097120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe64131180 0x7fbe64097120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient: _finish_hunting 0 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 1 monclient: found mon.noname-a 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.681+0000 7fbe68e33700 1 -- 172.21.15.104:0/1736973357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64099080 con 0x7fbe50004860 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe68e33700 1 -- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 msgr2=0x7fbe64141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe68e33700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe64141340 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400eae0 tx=0x7fbe54009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 1 -- 172.21.15.104:0/1736973357 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5400c5d0 con 0x7fbe50004860 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: dump: 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:41.705 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: _finish_auth 0 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe6b097700 5 monclient: authenticate success, global_id 15129 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 1 -- 172.21.15.104:0/1736973357 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5400c730 con 0x7fbe50004860 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe6b097700 10 monclient: _renew_subs 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe6b097700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:41.706 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe64141880 con 0x7fbe50004860 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 1 -- 172.21.15.104:0/1736973357 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5400caf0 con 0x7fbe50004860 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe62ffd700 4 set_mon_vals no callback set 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe60ff9700 10 monclient: dump: 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:41.707 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe6b097700 10 monclient: _renew_subs 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe6b097700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.685+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6411e860 con 0x7fbe50004860 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe6b097700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe6b097700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe6b097700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64099800 con 0x7fbe50004860 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe60ff9700 1 -- 172.21.15.104:0/1736973357 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fbe540070d0 con 0x7fbe50004860 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe60ff9700 1 --2- 172.21.15.104:0/1736973357 >> v2:172.21.15.174:6812/17603 conn(0x7fbe44035a80 0x7fbe44037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe60ff9700 1 -- 172.21.15.104:0/1736973357 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbe54048110 con 0x7fbe50004860 2024-09-18T14:27:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe63fff700 1 --2- 172.21.15.104:0/1736973357 >> v2:172.21.15.174:6812/17603 conn(0x7fbe44035a80 0x7fbe44037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:41.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.689+0000 7fbe63fff700 10 monclient: get_auth_request con 0x7fbe44035a80 auth_method 0 2024-09-18T14:27:41.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.693+0000 7fbe63fff700 1 --2- 172.21.15.104:0/1736973357 >> v2:172.21.15.174:6812/17603 conn(0x7fbe44035a80 0x7fbe44037f30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009650 tx=0x7fbe4c01a000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:41.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.697+0000 7fbe60ff9700 1 -- 172.21.15.104:0/1736973357 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbe64099800 con 0x7fbe50004860 2024-09-18T14:27:41.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.697+0000 7fbe60ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:41.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.697+0000 7fbe60ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.933+0000 7fbe6b097700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:27:41.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.933+0000 7fbe6b097700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:41.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.933+0000 7fbe6b097700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:41.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.933+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe64099800 con 0x7fbe50004860 2024-09-18T14:27:41.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.933+0000 7fbe60ff9700 1 -- 172.21.15.104:0/1736973357 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fbe64099800 con 0x7fbe50004860 2024-09-18T14:27:41.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.933+0000 7fbe60ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:41.939 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.933+0000 7fbe60ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:41.940 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":98,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 >> v2:172.21.15.174:6812/17603 conn(0x7fbe44035a80 msgr2=0x7fbe44037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 --2- 172.21.15.104:0/1736973357 >> v2:172.21.15.174:6812/17603 conn(0x7fbe44035a80 0x7fbe44037f30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009650 tx=0x7fbe4c01a000 comp rx=0 tx=0).stop 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 10 monclient: shutdown 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 msgr2=0x7fbe64141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe64141340 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400eae0 tx=0x7fbe54009040 comp rx=0 tx=0).stop 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 shutdown_connections 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 --2- 172.21.15.104:0/1736973357 >> v2:172.21.15.174:6812/17603 conn(0x7fbe44035a80 0x7fbe44037f30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbe64131550 0x7fbe64097660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe64131180 0x7fbe64097120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 --2- 172.21.15.104:0/1736973357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe50004860 0x7fbe64141340 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 >> 172.21.15.104:0/1736973357 conn(0x7fbe640a3420 msgr2=0x7fbe64134ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 shutdown_connections 2024-09-18T14:27:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:41.937+0000 7fbe6b097700 1 -- 172.21.15.104:0/1736973357 wait complete. 2024-09-18T14:27:44.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:27:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.057+0000 7f1024f91700 10 monclient: get_monmap_and_config 2024-09-18T14:27:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.057+0000 7f1024f91700 10 monclient: build_initial_monmap 2024-09-18T14:27:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient: monmap: 2024-09-18T14:27:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:45.065 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:45.063651+0000 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:45.063651+0000 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:45.066 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 1 Processor -- start 2024-09-18T14:27:45.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 1 -- start start 2024-09-18T14:27:45.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient: init 2024-09-18T14:27:45.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:45.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10200a59d0 0x7f102009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient(hunting): picked mon.noname-a con 0x7f10200a59d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f102009bea0 0x7f102009f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient(hunting): picked mon.noname-c con 0x7f102009bea0 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f102009fcb0 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102009f780 con 0x7f102009fcb0 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102009af10 con 0x7f10200a59d0 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10200a1870 con 0x7f102009bea0 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:45.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f1024f91700 10 monclient(hunting): authenticate will time out at 1186.053614s 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10200a59d0 0x7f102009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10200a59d0 0x7f102009ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59902/0 (socket says 172.21.15.104:59902) 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 1 -- 172.21.15.104:0/2064070465 learned_addr learned my addr 172.21.15.104:0/2064070465 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 10 monclient(hunting): get_auth_request con 0x7f10200a59d0 auth_method 0 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 1 --2- 172.21.15.104:0/2064070465 >> v2:172.21.15.174:3300/0 conn(0x7f102009bea0 0x7f102009f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): get_auth_request con 0x7f102009bea0 auth_method 0 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient(hunting): handle_auth_done global_id 5794 payload 274 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 1 -- 172.21.15.104:0/2064070465 >> v2:172.21.15.104:3300/0 conn(0x7f10200a59d0 msgr2=0x7f102009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 1 --2- 172.21.15.104:0/2064070465 >> v2:172.21.15.104:3300/0 conn(0x7f10200a59d0 0x7f102009ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 1 -- 172.21.15.104:0/2064070465 >> v1:172.21.15.104:6789/0 conn(0x7f102009fcb0 legacy=0x7f10200a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient: _finish_hunting 0 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 1 monclient: found mon.noname-c 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:27:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.061+0000 7f101effd700 1 -- 172.21.15.104:0/2064070465 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10200a5da0 con 0x7f102009bea0 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101effd700 1 -- 172.21.15.104:0/2064070465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 msgr2=0x7f102009f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101effd700 1 --2- 172.21.15.104:0/2064070465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 0x7f102009f240 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f102006f350 tx=0x7f1010009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b880aee42eb445ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 1 -- 172.21.15.104:0/2064070465 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101000e070 con 0x7f102009bea0 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: dump: 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 5 monclient: authenticate success, global_id 5794 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 1 -- 172.21.15.104:0/2064070465 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101000bd70 con 0x7f102009bea0 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 4 set_mon_vals no callback set 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 1 -- 172.21.15.104:0/2064070465 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010012660 con 0x7f102009bea0 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f101e7fc700 10 monclient: dump: 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient: shutdown 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- 172.21.15.104:0/2064070465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 msgr2=0x7f102009f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 --2- 172.21.15.104:0/2064070465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 0x7f102009f240 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f102006f350 tx=0x7f1010009780 comp rx=0 tx=0).stop 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- 172.21.15.104:0/2064070465 shutdown_connections 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 --2- 172.21.15.104:0/2064070465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 0x7f102009f240 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 --2- 172.21.15.104:0/2064070465 >> v2:172.21.15.104:3300/0 conn(0x7f10200a59d0 0x7f102009ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- 172.21.15.104:0/2064070465 >> 172.21.15.104:0/2064070465 conn(0x7f1020067c10 msgr2=0x7f10200a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- 172.21.15.104:0/2064070465 shutdown_connections 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- 172.21.15.104:0/2064070465 wait complete. 2024-09-18T14:27:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient: build_initial_monmap 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient: monmap: 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 Processor -- start 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- start start 2024-09-18T14:27:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient: init 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 0x7f1020131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): picked mon.noname-b con 0x7f102009bea0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 0x7f102011b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): picked mon.noname-a con 0x7f102009fcb0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 0x7f102011be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): picked mon.noname-c con 0x7f10200a59d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102011f080 con 0x7f102009fcb0 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020134cc0 con 0x7f102009bea0 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102011eda0 con 0x7f10200a59d0 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.065+0000 7f1024f91700 10 monclient(hunting): authenticate will time out at 1186.059815s 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 0x7f102011b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 0x7f102011b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59912/0 (socket says 172.21.15.104:59912) 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 -- 172.21.15.104:0/1573527541 learned_addr learned my addr 172.21.15.104:0/1573527541 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): get_auth_request con 0x7f102009fcb0 auth_method 0 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 0x7f102011be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 0x7f1020131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 10 monclient(hunting): get_auth_request con 0x7f10200a59d0 auth_method 0 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 10 monclient(hunting): get_auth_request con 0x7f102009bea0 auth_method 0 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 1 -- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 msgr2=0x7f102011be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 1 -- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 msgr2=0x7f102011be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:45.082 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 0x7f102011be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient(hunting): handle_auth_done global_id 15135 payload 824 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 -- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 msgr2=0x7f102011be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101ffff700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 0x7f102011be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 0x7f102011be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 -- 172.21.15.104:0/1573527541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 msgr2=0x7f1020131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 0x7f1020131c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient: _finish_hunting 0 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 monclient: found mon.noname-a 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.083 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 -- 172.21.15.104:0/1573527541 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1010009a20 con 0x7f102009fcb0 2024-09-18T14:27:45.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 -- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 msgr2=0x7f102011b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101effd700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 0x7f102011b920 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f101000b6c0 tx=0x7f1010018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 1 -- 172.21.15.104:0/1573527541 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010012e70 con 0x7f102009fcb0 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: dump: 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.086 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: _finish_auth 0 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f1024f91700 5 monclient: authenticate success, global_id 15135 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 1 -- 172.21.15.104:0/1573527541 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010026460 con 0x7f102009fcb0 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f1024f91700 10 monclient: _renew_subs 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f1024f91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1020141f50 con 0x7f102009fcb0 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 1 -- 172.21.15.104:0/1573527541 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010026820 con 0x7f102009fcb0 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f10177fe700 10 monclient: dump: 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f1024f91700 10 monclient: _renew_subs 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f101dffb700 4 set_mon_vals no callback set 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f1024f91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.069+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102011dfe0 con 0x7f102009fcb0 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f10177fe700 1 -- 172.21.15.104:0/1573527541 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f1010026c10 con 0x7f102009fcb0 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f10177fe700 1 --2- 172.21.15.104:0/1573527541 >> v2:172.21.15.174:6812/17603 conn(0x7f1000035a80 0x7f1000037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f1024f91700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f1024f91700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f1024f91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102009f780 con 0x7f102009fcb0 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f10177fe700 1 -- 172.21.15.104:0/1573527541 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1010049f10 con 0x7f102009fcb0 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f101f7fe700 1 --2- 172.21.15.104:0/1573527541 >> v2:172.21.15.174:6812/17603 conn(0x7f1000035a80 0x7f1000037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.073+0000 7f101f7fe700 10 monclient: get_auth_request con 0x7f1000035a80 auth_method 0 2024-09-18T14:27:45.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.077+0000 7f101f7fe700 1 --2- 172.21.15.104:0/1573527541 >> v2:172.21.15.174:6812/17603 conn(0x7f1000035a80 0x7f1000037f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1008006d40 tx=0x7f1008009250 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.081+0000 7f10177fe700 1 -- 172.21.15.104:0/1573527541 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f102009f780 con 0x7f102009fcb0 2024-09-18T14:27:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.081+0000 7f10177fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:45.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.081+0000 7f10177fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:27:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f102009f780 con 0x7f102009fcb0 2024-09-18T14:27:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f10177fe700 1 -- 172.21.15.104:0/1573527541 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f102009f780 con 0x7f102009fcb0 2024-09-18T14:27:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f10177fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f10177fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:45.314 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 >> v2:172.21.15.174:6812/17603 conn(0x7f1000035a80 msgr2=0x7f1000037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 --2- 172.21.15.104:0/1573527541 >> v2:172.21.15.174:6812/17603 conn(0x7f1000035a80 0x7f1000037f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1008006d40 tx=0x7f1008009250 comp rx=0 tx=0).stop 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 10 monclient: shutdown 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 msgr2=0x7f102011b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 0x7f102011b920 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f101000b6c0 tx=0x7f1010018000 comp rx=0 tx=0).stop 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 shutdown_connections 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 --2- 172.21.15.104:0/1573527541 >> v2:172.21.15.174:6812/17603 conn(0x7f1000035a80 0x7f1000037f30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f10200a59d0 0x7f102011be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f102009fcb0 0x7f102011b920 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 --2- 172.21.15.104:0/1573527541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f102009bea0 0x7f1020131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 >> 172.21.15.104:0/1573527541 conn(0x7f1020067c10 msgr2=0x7f1020134ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 shutdown_connections 2024-09-18T14:27:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.309+0000 7f1024f91700 1 -- 172.21.15.104:0/1573527541 wait complete. 2024-09-18T14:27:45.327 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T14:27:45.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T14:27:45.328 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T14:27:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient: get_monmap_and_config 2024-09-18T14:27:45.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient: build_initial_monmap 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient: monmap: 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:45.429138+0000 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:45.429138+0000 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:45.431 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:45.432 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 Processor -- start 2024-09-18T14:27:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 -- start start 2024-09-18T14:27:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient: init 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc33c097d80 0x7fc33c0952a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): picked mon.noname-a con 0x7fc33c097d80 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c098aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): picked mon.noname-b con 0x7fc33c0986d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc33c098fe0 0x7fc33c135380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): picked mon.noname-c con 0x7fc33c098fe0 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33c099a90 con 0x7fc33c097d80 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33c0957e0 con 0x7fc33c098fe0 2024-09-18T14:27:45.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c09aea0 con 0x7fc33c0986d0 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc34233a700 10 monclient(hunting): authenticate will time out at 1186.419200s 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc33c097d80 0x7fc33c0952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c098aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc33c097d80 0x7fc33c0952a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59914/0 (socket says 172.21.15.104:59914) 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c098aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:57936/0 (socket says 172.21.15.104:57936) 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 1 -- 172.21.15.104:0/2149421475 learned_addr learned my addr 172.21.15.104:0/2149421475 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 10 monclient(hunting): get_auth_request con 0x7fc33c097d80 auth_method 0 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 10 monclient(hunting): get_auth_request con 0x7fc33c0986d0 auth_method 0 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 1 -- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 msgr2=0x7fc33c098aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 1 -- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 msgr2=0x7fc33c098aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 1 --2- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c098aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.425+0000 7fc340b37700 1 --2- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c098aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 10 monclient(hunting): handle_auth_done global_id 15138 payload 274 2024-09-18T14:27:45.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 -- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 msgr2=0x7fc33c098aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 --2- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c098aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 -- 172.21.15.104:0/2149421475 >> v2:172.21.15.174:3300/0 conn(0x7fc33c098fe0 msgr2=0x7fc33c135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 --2- 172.21.15.104:0/2149421475 >> v2:172.21.15.174:3300/0 conn(0x7fc33c098fe0 0x7fc33c135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 10 monclient: _finish_hunting 0 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 monclient: found mon.noname-a 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 -- 172.21.15.104:0/2149421475 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c09a240 con 0x7fc33c097d80 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 -- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c097d80 msgr2=0x7fc33c0952a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc341338700 1 --2- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c097d80 0x7fc33c0952a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc33c09a980 tx=0x7fc330009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fdd4e98953613b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 1 -- 172.21.15.104:0/2149421475 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33000ba70 con 0x7fc33c097d80 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: dump: 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.442 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: _finish_auth 0 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 5 monclient: authenticate success, global_id 15138 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 1 -- 172.21.15.104:0/2149421475 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33000bbd0 con 0x7fc33c097d80 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 1 -- 172.21.15.104:0/2149421475 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330011420 con 0x7fc33c097d80 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 4 set_mon_vals no callback set 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:45.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc32ffff700 10 monclient: dump: 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 10 monclient: shutdown 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 1 -- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c097d80 msgr2=0x7fc33c0952a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.429+0000 7fc34233a700 1 --2- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c097d80 0x7fc33c0952a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc33c09a980 tx=0x7fc330009250 comp rx=0 tx=0).stop 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- 172.21.15.104:0/2149421475 shutdown_connections 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 --2- 172.21.15.104:0/2149421475 >> v2:172.21.15.174:3300/0 conn(0x7fc33c098fe0 0x7fc33c135380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 --2- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c098aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 --2- 172.21.15.104:0/2149421475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c097d80 0x7fc33c0952a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- 172.21.15.104:0/2149421475 >> 172.21.15.104:0/2149421475 conn(0x7fc33c0a11f0 msgr2=0x7fc33c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- 172.21.15.104:0/2149421475 shutdown_connections 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- 172.21.15.104:0/2149421475 wait complete. 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient: build_initial_monmap 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient: monmap: 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:45.445 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 Processor -- start 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- start start 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient: init 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:27:45.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc33c097d80 0x7fc33c09c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): picked mon.noname-b con 0x7fc33c097d80 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c12e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): picked mon.noname-c con 0x7fc33c0986d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c098fe0 0x7fc33c1286a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): picked mon.noname-a con 0x7fc33c098fe0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c12ab40 con 0x7fc33c098fe0 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c12ae20 con 0x7fc33c097d80 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c12b100 con 0x7fc33c0986d0 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc34233a700 10 monclient(hunting): authenticate will time out at 1186.426411s 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c12e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c098fe0 0x7fc33c1286a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c12e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:57938/0 (socket says 172.21.15.104:57938) 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 1 -- 172.21.15.104:0/1276132154 learned_addr learned my addr 172.21.15.104:0/1276132154 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 10 monclient(hunting): get_auth_request con 0x7fc33c0986d0 auth_method 0 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): get_auth_request con 0x7fc33c098fe0 auth_method 0 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 msgr2=0x7fc33c12e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:27:45.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 msgr2=0x7fc33c12e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c12e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc33c097d80 0x7fc33c09c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc340b37700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c12e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 10 monclient(hunting): get_auth_request con 0x7fc33c097d80 auth_method 0 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient(hunting): handle_auth_done global_id 15141 payload 824 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 msgr2=0x7fc33c12e930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fc33c0986d0 0x7fc33c12e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc33c097d80 msgr2=0x7fc33c09c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc33c097d80 0x7fc33c09c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient: _finish_hunting 0 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 1 monclient: found mon.noname-a 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.433+0000 7fc341b39700 1 -- 172.21.15.104:0/1276132154 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3300094f0 con 0x7fc33c098fe0 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341b39700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c098fe0 msgr2=0x7fc33c1286a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341b39700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c098fe0 0x7fc33c1286a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc33400bad0 tx=0x7fc334009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33400ca50 con 0x7fc33c098fe0 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: dump: 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.453 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: _finish_auth 0 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33400cbb0 con 0x7fc33c098fe0 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 5 monclient: authenticate success, global_id 15141 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 10 monclient: _renew_subs 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33c09ce80 con 0x7fc33c098fe0 2024-09-18T14:27:45.454 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334014410 con 0x7fc33c098fe0 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 10 monclient: dump: 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.455 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 10 monclient: _renew_subs 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32f7fe700 4 set_mon_vals no callback set 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c129be0 con 0x7fc33c098fe0 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fc3340147c0 con 0x7fc33c098fe0 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 0x7fc33c128be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0686b0 con 0x7fc33c13f390 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 1 --2- 172.21.15.104:0/1276132154 >> v2:172.21.15.174:6812/17603 conn(0x7fc320035a80 0x7fc320037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc334047df0 con 0x7fc33c098fe0 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 0x7fc33c128be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient: get_auth_request con 0x7fc33c13f390 auth_method 0 2024-09-18T14:27:45.456 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc340b37700 1 --2- 172.21.15.104:0/1276132154 >> v2:172.21.15.174:6812/17603 conn(0x7fc320035a80 0x7fc320037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 10 monclient(hunting): handle_auth_done global_id 15144 payload 824 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc341338700 1 -- 172.21.15.104:0/1276132154 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc33c12acb0 con 0x7fc33c13f390 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.437+0000 7fc340b37700 10 monclient: get_auth_request con 0x7fc320035a80 auth_method 0 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc341338700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 msgr2=0x7fc33c128be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc341338700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 0x7fc33c128be0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc330013000 tx=0x7fc330017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330020070 con 0x7fc33c13f390 2024-09-18T14:27:45.457 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 10 monclient: dump: 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc340b37700 1 --2- 172.21.15.104:0/1276132154 >> v2:172.21.15.174:6812/17603 conn(0x7fc320035a80 0x7fc320037f30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc33c06f170 tx=0x7fc338008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc330011e00 con 0x7fc33c13f390 2024-09-18T14:27:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 msgr2=0x7fc33c128be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 0x7fc33c128be0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc330013000 tx=0x7fc330017000 comp rx=0 tx=0).stop 2024-09-18T14:27:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 msgr2=0x7fc33c128be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:27:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.441+0000 7fc32d7fa700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c13f390 0x7fc33c128be0 secure :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc330013000 tx=0x7fc330017000 comp rx=0 tx=0).stop 2024-09-18T14:27:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc34233a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:27:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc34233a700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 0x7fc33c129120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc34233a700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c063450 con 0x7fc33c12ee70 2024-09-18T14:27:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 0x7fc33c129120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient: get_auth_request con 0x7fc33c12ee70 auth_method 0 2024-09-18T14:27:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 10 monclient(hunting): handle_auth_done global_id 15147 payload 824 2024-09-18T14:27:45.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 1 -- 172.21.15.104:0/1276132154 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc33c0a08a0 con 0x7fc33c12ee70 2024-09-18T14:27:45.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.453+0000 7fc341338700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 msgr2=0x7fc33c129120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc341338700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 0x7fc33c129120 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc330009290 tx=0x7fc33001d050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330026bc0 con 0x7fc33c12ee70 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 10 monclient: dump: 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.463 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 <== mon.0 v2:172.21.15.104:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fc33001a420 con 0x7fc33c12ee70 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 msgr2=0x7fc33c129120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 0x7fc33c129120 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc330009290 tx=0x7fc33001d050 comp rx=0 tx=0).stop 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 msgr2=0x7fc33c129120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc32d7fa700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c12ee70 0x7fc33c129120 secure :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc330009290 tx=0x7fc33001d050 comp rx=0 tx=0).stop 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:45.464 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "leader", 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 32, 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 0, 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:27:45.465 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:45.466 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:27:45.467 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:27:45.468 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.469 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:45.470 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:27:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:45.472 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:45.473 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.474 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:27:45.475 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 2 2024-09-18T14:27:45.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:45.477 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:45.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:45.477 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:27:45.477 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:45.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 >> v2:172.21.15.174:6812/17603 conn(0x7fc320035a80 msgr2=0x7fc320037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 --2- 172.21.15.104:0/1276132154 >> v2:172.21.15.174:6812/17603 conn(0x7fc320035a80 0x7fc320037f30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc33c06f170 tx=0x7fc338008040 comp rx=0 tx=0).stop 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 10 monclient: shutdown 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c098fe0 msgr2=0x7fc33c1286a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 --2- 172.21.15.104:0/1276132154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc33c098fe0 0x7fc33c1286a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc33400bad0 tx=0x7fc334009780 comp rx=0 tx=0).stop 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc341b39700 1 -- 172.21.15.104:0/1276132154 reap_dead start 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc341b39700 1 -- 172.21.15.104:0/1276132154 reap_dead start 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 shutdown_connections 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 >> 172.21.15.104:0/1276132154 conn(0x7fc33c0a11f0 msgr2=0x7fc33c0967e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 shutdown_connections 2024-09-18T14:27:45.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.457+0000 7fc34233a700 1 -- 172.21.15.104:0/1276132154 wait complete. 2024-09-18T14:27:45.479 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:27:45.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T14:27:45.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.577+0000 7f830ffff700 10 monclient: get_monmap_and_config 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.577+0000 7f830ffff700 10 monclient: build_initial_monmap 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.577+0000 7f830ffff700 10 monclient: monmap: 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:45.583457+0000 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:45.583457+0000 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.585 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:45.586 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:45.586 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:45.586 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:45.586 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:45.586 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 1 Processor -- start 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 1 -- start start 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient: init 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8310097d80 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83100952a0 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f8310137770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f8310135390 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8310099a90 con 0x7f8310097d80 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831009aea0 con 0x7f83100952a0 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83100994f0 con 0x7f8310135390 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830effd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8310097d80 0x7f83100986d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:49410/0 (socket says 172.21.15.104:49410) 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830effd700 1 -- 172.21.15.104:0/3934612014 learned_addr learned my addr 172.21.15.104:0/3934612014 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f8310137770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830ffff700 10 monclient(hunting): authenticate will time out at 1186.573968s 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 10 monclient(hunting): get_auth_request con 0x7f8310135390 auth_method 0 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 1 -- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 msgr2=0x7f8310137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:27:45.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 1 -- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 msgr2=0x7f8310137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:27:45.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f8310137770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:45.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.581+0000 7f830f7fe700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f8310137770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 -- 172.21.15.104:0/3934612014 >> v1:172.21.15.104:6789/0 conn(0x7f8310097d80 legacy=0x7f83100986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 1 -- 172.21.15.104:0/3934612014 >> v1:172.21.15.174:6789/0 conn(0x7f83100952a0 legacy=0x7f8310098de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812218977 0 0) 0x7f8310099a90 con 0x7f8310097d80 2024-09-18T14:27:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): my global_id is 15150 2024-09-18T14:27:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82fc003570 con 0x7f8310097d80 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296085112 0 0) 0x7f831009aea0 con 0x7f83100952a0 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): my global_id is 5803 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8310099a90 con 0x7f83100952a0 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3798165652 0 0) 0x7f82fc003570 con 0x7f8310097d80 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 msgr2=0x7f8310137770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f8310137770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 >> v1:172.21.15.174:6789/0 conn(0x7f83100952a0 legacy=0x7f8310098de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: _finish_hunting 0 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 monclient: found mon.noname-a-legacy 2024-09-18T14:27:45.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.104:6789/0 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831009a240 con 0x7f8310097d80 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830ffff700 5 monclient: authenticate success, global_id 15150 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4057020423 0 0) 0x7f82f80017a0 con 0x7f8310097d80 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: dump: 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.597 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] but i'm connected to v1:172.21.15.104:6789/0, reconnecting 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 >> v1:172.21.15.104:6789/0 conn(0x7f8310097d80 legacy=0x7f83100986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f82fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): picked mon.c con 0x7f8310135390 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82fc004340 0x7f82fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): picked mon.a con 0x7f82fc004340 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82fc004f30 0x7f82fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): picked mon.b con 0x7f82fc004f30 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83100994f0 con 0x7f82fc004340 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0056a0 con 0x7f82fc004f30 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc00bff0 con 0x7f8310135390 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830dffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T14:27:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830ffff700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82fc004340 0x7f82fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f82fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 10 monclient(hunting): get_auth_request con 0x7f82fc004340 auth_method 0 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): get_auth_request con 0x7f8310135390 auth_method 0 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830f7fe700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82fc004f30 0x7f82fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830f7fe700 10 monclient(hunting): get_auth_request con 0x7f82fc004f30 auth_method 0 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient(hunting): handle_auth_done global_id 15150 payload 293 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 -- 172.21.15.104:0/3934612014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82fc004f30 msgr2=0x7f82fc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82fc004f30 0x7f82fc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 -- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82fc004340 msgr2=0x7f82fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:45.601 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82fc004340 0x7f82fc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient: _finish_hunting 0 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 monclient: found mon.c 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 1 -- 172.21.15.104:0/3934612014 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc003570 con 0x7f8310135390 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 10 monclient: _finish_auth 0 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.585+0000 7f830effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830effd700 1 -- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 msgr2=0x7f82fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830effd700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f82fc003d90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c5c0 tx=0x7f831009a3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc0550c611e2c187 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f800ce50 con 0x7f8310135390 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 10 monclient: dump: 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8005920 con 0x7f8310135390 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:45.603 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 1 -- 172.21.15.104:0/3934612014 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8005ce0 con 0x7f8310135390 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 4 set_mon_vals no callback set 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830dffb700 10 monclient: dump: 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.604 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 10 monclient: shutdown 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 -- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 msgr2=0x7f82fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 --2- 172.21.15.104:0/3934612014 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310135390 0x7f82fc003d90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c5c0 tx=0x7f831009a3d0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830f7fe700 1 -- 172.21.15.104:0/3934612014 reap_dead start 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 -- 172.21.15.104:0/3934612014 shutdown_connections 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 -- 172.21.15.104:0/3934612014 >> 172.21.15.104:0/3934612014 conn(0x7f83100a11f0 msgr2=0x7f83100a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 -- 172.21.15.104:0/3934612014 shutdown_connections 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 -- 172.21.15.104:0/3934612014 wait complete. 2024-09-18T14:27:45.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 10 monclient: build_initial_monmap 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 10 monclient: monmap: 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:45.606 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:45.607 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 Processor -- start 2024-09-18T14:27:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 1 -- start start 2024-09-18T14:27:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.589+0000 7f830ffff700 10 monclient: init 2024-09-18T14:27:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:27:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83101345d0 0x7f83101349a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f83101345d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 0x7f831008ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f8310134ee0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f831008d350 0x7f831008d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f831008d350 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83100939b0 con 0x7f83101345d0 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8310099f10 con 0x7f831008d350 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8310094740 con 0x7f8310134ee0 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient(hunting): authenticate will time out at 1186.584310s 2024-09-18T14:27:45.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 0x7f831008ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 0x7f831008ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:57956/0 (socket says 172.21.15.104:57956) 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 learned_addr learned my addr 172.21.15.104:0/2709440033 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 10 monclient(hunting): get_auth_request con 0x7f8310134ee0 auth_method 0 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83101345d0 0x7f83101349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 msgr2=0x7f831008ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 msgr2=0x7f831008ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 0x7f831008ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): get_auth_request con 0x7f83101345d0 auth_method 0 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 0x7f831008ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830f7fe700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f831008d350 0x7f831008d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830f7fe700 10 monclient(hunting): get_auth_request con 0x7f831008d350 auth_method 0 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient(hunting): handle_auth_done global_id 15153 payload 824 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 msgr2=0x7f831008ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8310134ee0 0x7f831008ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f831008d350 msgr2=0x7f831008d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f831008d350 0x7f831008d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient: _finish_hunting 0 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 monclient: found mon.noname-a 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831008e870 con 0x7f83101345d0 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830f7fe700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f831008d350 0x7f831008d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83101345d0 msgr2=0x7f83101349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830effd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83101345d0 0x7f83101349a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f82f8014000 tx=0x7f82f8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f80072e0 con 0x7f83101345d0 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:45.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: dump: 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: _finish_auth 0 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 5 monclient: authenticate success, global_id 15153 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8007440 con 0x7f83101345d0 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient: _renew_subs 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83100903a0 con 0x7f83101345d0 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f80077e0 con 0x7f83101345d0 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f8306ffd700 10 monclient: dump: 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830d7fa700 4 set_mon_vals no callback set 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient: _renew_subs 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.593+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831008f440 con 0x7f83101345d0 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830ffff700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83100686b0 con 0x7f831002ae80 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f831002ae80 auth_method 0 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f82f8019070 con 0x7f83101345d0 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f8306ffd700 1 --2- 172.21.15.104:0/2709440033 >> v2:172.21.15.174:6812/17603 conn(0x7f82f0035a80 0x7f82f0037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:45.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f82f80490e0 con 0x7f83101345d0 2024-09-18T14:27:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830f7fe700 1 --2- 172.21.15.104:0/2709440033 >> v2:172.21.15.174:6812/17603 conn(0x7f82f0035a80 0x7f82f0037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.597+0000 7f830f7fe700 10 monclient: get_auth_request con 0x7f82f0035a80 auth_method 0 2024-09-18T14:27:45.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.601+0000 7f830f7fe700 1 --2- 172.21.15.104:0/2709440033 >> v2:172.21.15.174:6812/17603 conn(0x7f82f0035a80 0x7f82f0037f30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f831006fe40 tx=0x7f8300008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f831002ae80 auth_method 0 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:45.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:45.801+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f831002ae80 auth_method 0 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:46.205+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f831002ae80 auth_method 0 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:47.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:47.009+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T14:27:48.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.593+0000 7f8305ffb700 10 monclient: tick 2024-09-18T14:27:48.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.593+0000 7f8305ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:48.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.593+0000 7f8305ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:48.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:48.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f831002ae80 auth_method 0 2024-09-18T14:27:48.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:48.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:48.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:48.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:48.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:48.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:48.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:48.613+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-18T14:27:49.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 10 monclient: tick 2024-09-18T14:27:49.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:49.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 5 monclient: _check_tell_commands timeout tell command 1 2024-09-18T14:27:49.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T14:27:49.598 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 msgr2=0x7f831008dd00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831002ae80 0x7f831008dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f8305ffb700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80060c0 con 0x7f82e8003590 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f82e8003590 auth_method 0 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 msgr2=0x7f82e8005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 msgr2=0x7f82e8005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:49.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.593+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f82e8003590 auth_method 0 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 msgr2=0x7f82e8005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 msgr2=0x7f82e8005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T14:27:49.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:49.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:49.797+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T14:27:50.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f82e8003590 auth_method 0 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e8006230 con 0x7f82e8003590 2024-09-18T14:27:50.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 msgr2=0x7f82e8005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8308007ec0 tx=0x7f8308009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830801c070 con 0x7f82e8003590 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f8306ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f8306ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.201+0000 7f8306ffd700 10 monclient: dump: 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.212 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.205+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8308021470 con 0x7f82e8003590 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.205+0000 7f8306ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.205+0000 7f8306ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.205+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 msgr2=0x7f82e8005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.205+0000 7f8306ffd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8308007ec0 tx=0x7f8308009040 comp rx=0 tx=0).stop 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.205+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 msgr2=0x7f82e8005a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:27:50.213 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.205+0000 7f8306ffd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f82e8003590 0x7f82e8005a10 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8308007ec0 tx=0x7f8308009040 comp rx=0 tx=0).stop 2024-09-18T14:27:50.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:27:50.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830ffff700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 0x7f831005ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.224 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8310090530 con 0x7f831005e890 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 0x7f831005ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient: get_auth_request con 0x7f831005e890 auth_method 0 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.225 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-09-18T14:27:50.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.217+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f831006ddd0 con 0x7f831005e890 2024-09-18T14:27:50.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f830e7fc700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 msgr2=0x7f831005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f830e7fc700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 0x7f831005ec90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8308020000 tx=0x7f8308021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308015d90 con 0x7f831005e890 2024-09-18T14:27:50.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:50.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 10 monclient: dump: 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.228 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 <== mon.2 v2:172.21.15.104:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f83080203d0 con 0x7f831005e890 2024-09-18T14:27:50.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:27:50.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:50.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 msgr2=0x7f831005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 0x7f831005ec90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8308020000 tx=0x7f8308021000 comp rx=0 tx=0).stop 2024-09-18T14:27:50.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f830f7fe700 1 -- 172.21.15.104:0/2709440033 reap_dead start 2024-09-18T14:27:50.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 msgr2=0x7f831005ec90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:27:50.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.221+0000 7f8306ffd700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f831005e890 0x7f831005ec90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.230 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:50.230 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:27:50.230 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:27:50.230 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 32, 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 4, 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:50.231 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:50.232 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:27:50.233 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:50.234 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.235 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.236 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.237 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.239 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:27:50.240 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.241 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:27:50.242 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:50.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 >> v2:172.21.15.174:6812/17603 conn(0x7f82f0035a80 msgr2=0x7f82f0037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 --2- 172.21.15.104:0/2709440033 >> v2:172.21.15.174:6812/17603 conn(0x7f82f0035a80 0x7f82f0037f30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f831006fe40 tx=0x7f8300008040 comp rx=0 tx=0).stop 2024-09-18T14:27:50.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 10 monclient: shutdown 2024-09-18T14:27:50.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:50.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83101345d0 msgr2=0x7f83101349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83101345d0 0x7f83101349a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f82f8014000 tx=0x7f82f8015000 comp rx=0 tx=0).stop 2024-09-18T14:27:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 shutdown_connections 2024-09-18T14:27:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 --2- 172.21.15.104:0/2709440033 >> v2:172.21.15.174:6812/17603 conn(0x7f82f0035a80 0x7f82f0037f30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 --2- 172.21.15.104:0/2709440033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83101345d0 0x7f83101349a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 >> 172.21.15.104:0/2709440033 conn(0x7f83100a11f0 msgr2=0x7f831006af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 shutdown_connections 2024-09-18T14:27:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.225+0000 7f830ffff700 1 -- 172.21.15.104:0/2709440033 wait complete. 2024-09-18T14:27:50.247 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:27:50.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T14:27:50.247 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T14:27:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.349+0000 7f55d359e700 10 monclient: get_monmap_and_config 2024-09-18T14:27:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.349+0000 7f55d359e700 10 monclient: build_initial_monmap 2024-09-18T14:27:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.349+0000 7f55d359e700 10 monclient: monmap: 2024-09-18T14:27:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.356 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:50.354793+0000 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:50.354793+0000 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:50.357 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.349+0000 7f55d359e700 1 Processor -- start 2024-09-18T14:27:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.349+0000 7f55d359e700 1 -- start start 2024-09-18T14:27:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.349+0000 7f55d359e700 10 monclient: init 2024-09-18T14:27:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:50.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d4097c10 0x7f55d4096300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient(hunting): picked mon.noname-b con 0x7f55d4097c10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55d409bbe0 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55d409c6c0 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d4095920 con 0x7f55d409bbe0 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d4097fe0 con 0x7f55d409c6c0 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d409cf10 con 0x7f55d4097c10 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d359e700 10 monclient(hunting): authenticate will time out at 1191.344438s 2024-09-18T14:27:50.360 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d4097c10 0x7f55d4096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55c9d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f55d409bbe0 0x7f55d409bfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34780/0 (socket says 172.21.15.104:34780) 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55c9d9b700 1 -- 172.21.15.104:0/4143855649 learned_addr learned my addr 172.21.15.104:0/4143855649 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): get_auth_request con 0x7f55d4097c10 auth_method 0 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient(hunting): handle_auth_done global_id 34106 payload 274 2024-09-18T14:27:50.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 1 -- 172.21.15.104:0/4143855649 >> v1:172.21.15.174:6789/0 conn(0x7f55d409c6c0 legacy=0x7f55d41373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:50.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 1 -- 172.21.15.104:0/4143855649 >> v1:172.21.15.104:6789/0 conn(0x7f55d409bbe0 legacy=0x7f55d409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:50.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient: _finish_hunting 0 2024-09-18T14:27:50.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 1 monclient: found mon.noname-b 2024-09-18T14:27:50.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:50.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:50.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 1 -- 172.21.15.104:0/4143855649 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d4095640 con 0x7f55d4097c10 2024-09-18T14:27:50.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 1 -- 172.21.15.104:0/4143855649 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d4097c10 msgr2=0x7f55d4096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d259c700 1 --2- 172.21.15.104:0/4143855649 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d4097c10 0x7f55d4096300 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009910 tx=0x7f55c40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a3ad06dade008d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.353+0000 7f55d1d9b700 1 -- 172.21.15.104:0/4143855649 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c400ba70 con 0x7f55d4097c10 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: dump: 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.364 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: _finish_auth 0 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 1 -- 172.21.15.104:0/4143855649 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c400bbd0 con 0x7f55d4097c10 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 1 -- 172.21.15.104:0/4143855649 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4011420 con 0x7f55d4097c10 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 5 monclient: authenticate success, global_id 34106 2024-09-18T14:27:50.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 4 set_mon_vals no callback set 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d1d9b700 10 monclient: dump: 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.366 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 10 monclient: shutdown 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 -- 172.21.15.104:0/4143855649 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d4097c10 msgr2=0x7f55d4096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 --2- 172.21.15.104:0/4143855649 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d4097c10 0x7f55d4096300 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009910 tx=0x7f55c40091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 -- 172.21.15.104:0/4143855649 shutdown_connections 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 --2- 172.21.15.104:0/4143855649 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d4097c10 0x7f55d4096300 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 -- 172.21.15.104:0/4143855649 >> 172.21.15.104:0/4143855649 conn(0x7f55d40a11f0 msgr2=0x7f55d40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 -- 172.21.15.104:0/4143855649 shutdown_connections 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 -- 172.21.15.104:0/4143855649 wait complete. 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 10 monclient: build_initial_monmap 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 10 monclient: monmap: 2024-09-18T14:27:50.367 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:50.368 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 Processor -- start 2024-09-18T14:27:50.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 1 -- start start 2024-09-18T14:27:50.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.357+0000 7f55d359e700 10 monclient: init 2024-09-18T14:27:50.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:50.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:27:50.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d4097c10 0x7f55d41414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.374 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): picked mon.noname-b con 0x7f55d4097c10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d409bbe0 0x7f55d4094360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): picked mon.noname-c con 0x7f55d409bbe0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d409c6c0 0x7f55d408dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): picked mon.noname-a con 0x7f55d409c6c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d409a280 con 0x7f55d409c6c0 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d4142350 con 0x7f55d4097c10 2024-09-18T14:27:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d4091580 con 0x7f55d409bbe0 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient(hunting): authenticate will time out at 1191.352235s 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d409bbe0 0x7f55d4094360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d409c6c0 0x7f55d408dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d409bbe0 0x7f55d4094360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42848/0 (socket says 172.21.15.104:42848) 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 1 -- 172.21.15.104:0/3704601832 learned_addr learned my addr 172.21.15.104:0/3704601832 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 10 monclient(hunting): get_auth_request con 0x7f55d409bbe0 auth_method 0 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): get_auth_request con 0x7f55d409c6c0 auth_method 0 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d4097c10 0x7f55d41414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 10 monclient(hunting): get_auth_request con 0x7f55d4097c10 auth_method 0 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55c9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient(hunting): handle_auth_done global_id 15156 payload 824 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d409bbe0 msgr2=0x7f55d4094360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f55d409bbe0 0x7f55d4094360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d4097c10 msgr2=0x7f55d41414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d4097c10 0x7f55d41414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 monclient: found mon.noname-a 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 -- 172.21.15.104:0/3704601832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4009420 con 0x7f55d409c6c0 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d409c6c0 msgr2=0x7f55d408dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d2d9d700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d409c6c0 0x7f55d408dee0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00c810 tx=0x7f55cc002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc012840 con 0x7f55d409c6c0 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:50.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: dump: 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.380 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: _finish_auth 0 2024-09-18T14:27:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 5 monclient: authenticate success, global_id 15156 2024-09-18T14:27:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient: _renew_subs 2024-09-18T14:27:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d4091860 con 0x7f55d409c6c0 2024-09-18T14:27:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc0129c0 con 0x7f55d409c6c0 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc012d80 con 0x7f55d409c6c0 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55caffd700 10 monclient: dump: 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient: _renew_subs 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d4141a20 con 0x7f55d409c6c0 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.361+0000 7f55d159a700 4 set_mon_vals no callback set 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d359e700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 0x7f55d4029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40686b0 con 0x7f55d40948a0 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 0x7f55d4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient: get_auth_request con 0x7f55d40948a0 auth_method 0 2024-09-18T14:27:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f55cc011590 con 0x7f55d409c6c0 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55caffd700 1 --2- 172.21.15.104:0/3704601832 >> v2:172.21.15.174:6812/17603 conn(0x7f55b4035a80 0x7f55b4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f55cc048140 con 0x7f55d409c6c0 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 10 monclient(hunting): handle_auth_done global_id 5812 payload 824 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55d259c700 1 -- 172.21.15.104:0/3704601832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55d406e6c0 con 0x7f55d40948a0 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55c9d9b700 1 --2- 172.21.15.104:0/3704601832 >> v2:172.21.15.174:6812/17603 conn(0x7f55b4035a80 0x7f55b4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.365+0000 7f55c9d9b700 10 monclient: get_auth_request con 0x7f55b4035a80 auth_method 0 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55d259c700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 msgr2=0x7f55d4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55d259c700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 0x7f55d4029d00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009910 tx=0x7f55c4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4021070 con 0x7f55d40948a0 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 10 monclient: dump: 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55c9d9b700 1 --2- 172.21.15.104:0/3704601832 >> v2:172.21.15.174:6812/17603 conn(0x7f55b4035a80 0x7f55b4037f30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f55d408f6c0 tx=0x7f55bc004a80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f55c401c660 con 0x7f55d40948a0 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 msgr2=0x7f55d4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 0x7f55d4029d00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009910 tx=0x7f55c4016000 comp rx=0 tx=0).stop 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 msgr2=0x7f55d4029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:27:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.369+0000 7f55caffd700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d40948a0 0x7f55d4029d00 secure :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009910 tx=0x7f55c4016000 comp rx=0 tx=0).stop 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d359e700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 0x7f55d402a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d4094c70 con 0x7f55d402a240 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 0x7f55d402a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 10 monclient: get_auth_request con 0x7f55d402a240 auth_method 0 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.377+0000 7f55d259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55d259c700 10 monclient(hunting): handle_auth_done global_id 5815 payload 824 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55d259c700 1 -- 172.21.15.104:0/3704601832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f55d4063450 con 0x7f55d402a240 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55d259c700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 msgr2=0x7f55d402a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55d259c700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 0x7f55d402a610 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009860 tx=0x7f55c401e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4032470 con 0x7f55d402a240 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 10 monclient: dump: 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.390 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 <== mon.1 v2:172.21.15.174:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f55c401c450 con 0x7f55d402a240 2024-09-18T14:27:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T14:27:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 msgr2=0x7f55d402a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 0x7f55d402a610 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009860 tx=0x7f55c401e070 comp rx=0 tx=0).stop 2024-09-18T14:27:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 msgr2=0x7f55d402a610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T14:27:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.381+0000 7f55caffd700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55d402a240 0x7f55d402a610 secure :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009860 tx=0x7f55c401e070 comp rx=0 tx=0).stop 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "peon", 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: "election_epoch": 32, 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum": [ 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: 0, 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: 1, 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: 2 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_age": 5, 2024-09-18T14:27:50.392 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "required_con": "2449958755906961412", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "required_mon": [ 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.393 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_con": "4540138320759226367", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "quorum_mon": [ 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "outside_quorum": [], 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "extra_probe_peers": [], 2024-09-18T14:27:50.394 INFO:teuthology.orchestra.run.smithi104.stdout: "sync_provider": [], 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "monmap": { 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "epoch": 1, 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "fsid": "ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8", 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "modified": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "created": "2024-09-18T14:21:05.806012Z", 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release": 17, 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "election_strategy": 1, 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "disallowed_leaders: ": "", 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false, 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "tiebreaker_mon": "", 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "removed_ranks: ": "", 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "features": { 2024-09-18T14:27:50.395 INFO:teuthology.orchestra.run.smithi104.stdout: "persistent": [ 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "kraken", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "luminous", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "mimic", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap-prune", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "nautilus", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "octopus", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "pacific", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "elector-pinging", 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "quincy" 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "optional": [] 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: "mons": [ 2024-09-18T14:27:50.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 0, 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "a", 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3300", 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789", 2024-09-18T14:27:50.397 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6789/0", 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6789/0", 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 1, 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "b", 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:50.398 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:3300", 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789", 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.174:6789/0", 2024-09-18T14:27:50.399 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.174:6789/0", 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "rank": 2, 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "c", 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addrs": { 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:3301", 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790", 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 0 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6790/0", 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "public_addr": "172.21.15.104:6790/0", 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "priority": 0, 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "weight": 0, 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: "crush_location": "{}" 2024-09-18T14:27:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: "feature_map": { 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": [ 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": [ 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.402 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 4 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: "client": [ 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:50.403 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 5 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": [ 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: "release": "luminous", 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 1 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout: "stretch_mode": false 2024-09-18T14:27:50.404 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:50.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 >> v2:172.21.15.174:6812/17603 conn(0x7f55b4035a80 msgr2=0x7f55b4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 --2- 172.21.15.104:0/3704601832 >> v2:172.21.15.174:6812/17603 conn(0x7f55b4035a80 0x7f55b4037f30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f55d408f6c0 tx=0x7f55bc004a80 comp rx=0 tx=0).stop 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 10 monclient: shutdown 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d409c6c0 msgr2=0x7f55d408dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 --2- 172.21.15.104:0/3704601832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d409c6c0 0x7f55d408dee0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00c810 tx=0x7f55cc002620 comp rx=0 tx=0).stop 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d2d9d700 1 -- 172.21.15.104:0/3704601832 reap_dead start 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d2d9d700 1 -- 172.21.15.104:0/3704601832 reap_dead start 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 shutdown_connections 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 >> 172.21.15.104:0/3704601832 conn(0x7f55d40a11f0 msgr2=0x7f55d409f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 shutdown_connections 2024-09-18T14:27:50.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.385+0000 7f55d359e700 1 -- 172.21.15.104:0/3704601832 wait complete. 2024-09-18T14:27:50.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T14:27:50.407 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T14:27:50.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.513+0000 7f95d65d1700 10 monclient: get_monmap_and_config 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.513+0000 7f95d65d1700 10 monclient: build_initial_monmap 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.513+0000 7f95d65d1700 10 monclient: monmap: 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:50.519588+0000 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:50.519588+0000 2024-09-18T14:27:50.521 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:50.522 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 1 Processor -- start 2024-09-18T14:27:50.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 1 -- start start 2024-09-18T14:27:50.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient: init 2024-09-18T14:27:50.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:50.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T14:27:50.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95d009d620 0x7f95d012eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.524 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient(hunting): picked mon.noname-a con 0x7f95d009d620 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95d012f080 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f480 0x7f95d00961f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient(hunting): picked mon.noname-b con 0x7f95d012f480 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d00990f0 con 0x7f95d012f080 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d009d9f0 con 0x7f95d009d620 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0096e10 con 0x7f95d012f480 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d65d1700 10 monclient(hunting): authenticate will time out at 1191.509217s 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95d009d620 0x7f95d012eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95d009d620 0x7f95d012eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35560/0 (socket says 172.21.15.104:35560) 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 -- 172.21.15.104:0/694738783 learned_addr learned my addr 172.21.15.104:0/694738783 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): get_auth_request con 0x7f95d009d620 auth_method 0 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d4b6e700 1 --2- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f480 0x7f95d00961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d4b6e700 10 monclient(hunting): get_auth_request con 0x7f95d012f480 auth_method 0 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d4b6e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d4b6e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95d4b6e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient(hunting): handle_auth_done global_id 15159 payload 274 2024-09-18T14:27:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 -- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f480 msgr2=0x7f95d00961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 --2- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f480 0x7f95d00961f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 -- 172.21.15.104:0/694738783 >> v1:172.21.15.174:6789/0 conn(0x7f95d012f080 legacy=0x7f95d0095ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient: _finish_hunting 0 2024-09-18T14:27:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 monclient: found mon.noname-a 2024-09-18T14:27:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.517+0000 7f95cffff700 1 -- 172.21.15.104:0/694738783 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d012f920 con 0x7f95d009d620 2024-09-18T14:27:50.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cffff700 1 -- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d009d620 msgr2=0x7f95d012eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cffff700 1 --2- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d009d620 0x7f95d012eb40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f95b80099e0 tx=0x7f95b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6f95d607f1436f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 1 -- 172.21.15.104:0/694738783 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b800ba30 con 0x7f95d009d620 2024-09-18T14:27:50.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: dump: 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: _finish_auth 0 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 5 monclient: authenticate success, global_id 15159 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 1 -- 172.21.15.104:0/694738783 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b800bb90 con 0x7f95d009d620 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 4 set_mon_vals no callback set 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 1 -- 172.21.15.104:0/694738783 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8011420 con 0x7f95d009d620 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95ceffd700 10 monclient: dump: 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient: shutdown 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d009d620 msgr2=0x7f95d012eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 --2- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d009d620 0x7f95d012eb40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f95b80099e0 tx=0x7f95b8009250 comp rx=0 tx=0).stop 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- 172.21.15.104:0/694738783 shutdown_connections 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 --2- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f480 0x7f95d00961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 --2- 172.21.15.104:0/694738783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d009d620 0x7f95d012eb40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- 172.21.15.104:0/694738783 >> 172.21.15.104:0/694738783 conn(0x7f95d00a1290 msgr2=0x7f95d00a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- 172.21.15.104:0/694738783 shutdown_connections 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- 172.21.15.104:0/694738783 wait complete. 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient: build_initial_monmap 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient: monmap: 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:50.535 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 Processor -- start 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- start start 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient: init 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d009d620 0x7f95d013e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): picked mon.noname-b con 0x7f95d009d620 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 0x7f95d012ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): picked mon.noname-c con 0x7f95d012f080 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d012f480 0x7f95d0141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): picked mon.noname-a con 0x7f95d012f480 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d009faf0 con 0x7f95d012f480 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d012c630 con 0x7f95d009d620 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d012c2f0 con 0x7f95d012f080 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d65d1700 10 monclient(hunting): authenticate will time out at 1191.515648s 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cf7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 0x7f95d012ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cf7fe700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 0x7f95d012ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42868/0 (socket says 172.21.15.104:42868) 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d4b6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d012f480 0x7f95d0141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cf7fe700 1 -- 172.21.15.104:0/460624401 learned_addr learned my addr 172.21.15.104:0/460624401 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cf7fe700 10 monclient(hunting): get_auth_request con 0x7f95d012f080 auth_method 0 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d4b6e700 10 monclient(hunting): get_auth_request con 0x7f95d012f480 auth_method 0 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d4b6e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d4b6e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.521+0000 7f95d4b6e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d009d620 0x7f95d013e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 10 monclient(hunting): get_auth_request con 0x7f95d009d620 auth_method 0 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d4b6e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d4b6e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d4b6e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 10 monclient(hunting): handle_auth_done global_id 34115 payload 824 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 -- 172.21.15.104:0/460624401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d009d620 msgr2=0x7f95d013e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d009d620 0x7f95d013e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 -- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d012f480 msgr2=0x7f95d0141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d012f480 0x7f95d0141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 monclient: found mon.noname-c 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 -- 172.21.15.104:0/460624401 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b80094f0 con 0x7f95d012f080 2024-09-18T14:27:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 -- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 msgr2=0x7f95d012ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cf7fe700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 0x7f95d012ddb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c5d0 tx=0x7f95c0007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 1 -- 172.21.15.104:0/460624401 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c00117a0 con 0x7f95d012f080 2024-09-18T14:27:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: dump: 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: _finish_auth 0 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 1 -- 172.21.15.104:0/460624401 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c0011900 con 0x7f95d012f080 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d65d1700 5 monclient: authenticate success, global_id 34115 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d65d1700 10 monclient: _renew_subs 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d65d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d006f380 con 0x7f95d012f080 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 1 -- 172.21.15.104:0/460624401 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c0011cc0 con 0x7f95d012f080 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95c7fff700 10 monclient: dump: 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95cdffb700 4 set_mon_vals no callback set 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d65d1700 10 monclient: _renew_subs 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d65d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.525+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d012b080 con 0x7f95d012f080 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95c7fff700 1 -- 172.21.15.104:0/460624401 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f95c0012630 con 0x7f95d012f080 2024-09-18T14:27:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95d65d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95d65d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95d65d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d0064b30 con 0x7f95d012f080 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95c7fff700 1 --2- 172.21.15.104:0/460624401 >> v2:172.21.15.174:6812/17603 conn(0x7f95b0035a80 0x7f95b0037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95c7fff700 1 -- 172.21.15.104:0/460624401 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f95c0047a20 con 0x7f95d012f080 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95cffff700 1 --2- 172.21.15.104:0/460624401 >> v2:172.21.15.174:6812/17603 conn(0x7f95b0035a80 0x7f95b0037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.529+0000 7f95cffff700 10 monclient: get_auth_request con 0x7f95b0035a80 auth_method 0 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.533+0000 7f95cffff700 1 --2- 172.21.15.104:0/460624401 >> v2:172.21.15.174:6812/17603 conn(0x7f95b0035a80 0x7f95b0037f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f95b80099e0 tx=0x7f95b8016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.537+0000 7f95c7fff700 1 -- 172.21.15.104:0/460624401 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f95c00108c0 con 0x7f95d012f080 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.537+0000 7f95c7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.537+0000 7f95c7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:50.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.757+0000 7f95d65d1700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T14:27:50.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.757+0000 7f95d65d1700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T14:27:50.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.757+0000 7f95d65d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:50.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.757+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f95d006f380 con 0x7f95d012f080 2024-09-18T14:27:50.769 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.761+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.769 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.761+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=7,logm=70} crc {auth=3610053680,config=3140514009,health=85192585,kv=1680675726,logm=409851358}) 2024-09-18T14:27:50.769 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.761+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=3610053680,config=3140514009,health=1017346500,kv=4078166765,logm=3175839122}) 2024-09-18T14:27:50.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.761+0000 7f95c7fff700 1 -- 172.21.15.104:0/460624401 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f95d006f380 con 0x7f95d012f080 2024-09-18T14:27:50.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.761+0000 7f95c7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T14:27:50.771 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.761+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.771 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.761+0000 7f95c7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:50.772 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.761+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=7,logm=70} crc {auth=3610053680,config=3140514009,health=85192585,kv=1680675726,logm=409851358}) 2024-09-18T14:27:50.772 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.761+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=7,logm=69} crc {auth=3610053680,config=3140514009,health=1017346500,kv=1053094754,logm=1282970367}) 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 >> v2:172.21.15.174:6812/17603 conn(0x7f95b0035a80 msgr2=0x7f95b0037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 --2- 172.21.15.104:0/460624401 >> v2:172.21.15.174:6812/17603 conn(0x7f95b0035a80 0x7f95b0037f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f95b80099e0 tx=0x7f95b8016000 comp rx=0 tx=0).stop 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 10 monclient: shutdown 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 msgr2=0x7f95d012ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 0x7f95d012ddb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c5d0 tx=0x7f95c0007f60 comp rx=0 tx=0).stop 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 shutdown_connections 2024-09-18T14:27:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 --2- 172.21.15.104:0/460624401 >> v2:172.21.15.174:6812/17603 conn(0x7f95b0035a80 0x7f95b0037f30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95d012f480 0x7f95d0141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f95d012f080 0x7f95d012ddb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 --2- 172.21.15.104:0/460624401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d009d620 0x7f95d013e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:50.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 >> 172.21.15.104:0/460624401 conn(0x7f95d00a1290 msgr2=0x7f95d0095a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:50.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 shutdown_connections 2024-09-18T14:27:50.774 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:50.765+0000 7f95d65d1700 1 -- 172.21.15.104:0/460624401 wait complete. 2024-09-18T14:27:50.782 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.782 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280807579}) 2024-09-18T14:27:50.782 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448958350}) 2024-09-18T14:27:50.782 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.782 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280807579}) 2024-09-18T14:27:50.782 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629609814}) 2024-09-18T14:27:50.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T14:27:50.785 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.785 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704617368}) 2024-09-18T14:27:50.785 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2821678173}) 2024-09-18T14:27:50.786 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.786 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704617368}) 2024-09-18T14:27:50.786 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.777+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3512741616}) 2024-09-18T14:27:50.786 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.787 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579782582}) 2024-09-18T14:27:50.787 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195427620}) 2024-09-18T14:27:50.787 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.787 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579782582}) 2024-09-18T14:27:50.787 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473874487}) 2024-09-18T14:27:50.788 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.788 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959630823}) 2024-09-18T14:27:50.789 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2740393293}) 2024-09-18T14:27:50.789 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.789 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959630823}) 2024-09-18T14:27:50.789 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.781+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229231676}) 2024-09-18T14:27:50.790 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.790 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024911785}) 2024-09-18T14:27:50.790 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3520024803}) 2024-09-18T14:27:50.790 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.790 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024911785}) 2024-09-18T14:27:50.790 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=118110669}) 2024-09-18T14:27:50.792 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.792 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431531499}) 2024-09-18T14:27:50.792 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=338422555}) 2024-09-18T14:27:50.792 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.792 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431531499}) 2024-09-18T14:27:50.792 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.785+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3167155248}) 2024-09-18T14:27:50.794 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.794 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858411162}) 2024-09-18T14:27:50.794 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565708731}) 2024-09-18T14:27:50.794 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.794 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858411162}) 2024-09-18T14:27:50.794 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925734181}) 2024-09-18T14:27:50.795 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.795 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248237471}) 2024-09-18T14:27:50.795 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185700189}) 2024-09-18T14:27:50.795 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.795 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248237471}) 2024-09-18T14:27:50.795 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3267068296}) 2024-09-18T14:27:50.796 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.796 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435554688}) 2024-09-18T14:27:50.797 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195422860}) 2024-09-18T14:27:50.797 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.797 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435554688}) 2024-09-18T14:27:50.797 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.789+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=632619512}) 2024-09-18T14:27:50.797 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.798 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10400927}) 2024-09-18T14:27:50.798 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171990109}) 2024-09-18T14:27:50.798 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.798 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10400927}) 2024-09-18T14:27:50.798 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574237394}) 2024-09-18T14:27:50.799 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.799 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611898210}) 2024-09-18T14:27:50.799 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629960818}) 2024-09-18T14:27:50.799 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.799 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611898210}) 2024-09-18T14:27:50.799 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878419003}) 2024-09-18T14:27:50.800 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.800 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989409708}) 2024-09-18T14:27:50.800 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884937643}) 2024-09-18T14:27:50.800 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.800 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989409708}) 2024-09-18T14:27:50.800 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.793+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321349003}) 2024-09-18T14:27:50.801 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.801 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1} crc {logm=1960380620,mdsmap=1531151339,mgr=1292546760,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=2364737929,mon_config_key=1784637070,monmap=530371891,osd_metadata=2735459733,osd_pg_creating=4181675810}) 2024-09-18T14:27:50.802 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=62,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=1313730356,mdsmap=3585761405,mgr=3617533101,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=1011741560,mon_config_key=1784637070,monmap=530371891,osd_metadata=2265752427,osd_pg_creating=4181675810,osd_snap=3178633393,osdmap=2722267019}) 2024-09-18T14:27:50.802 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.802 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1} crc {logm=1960380620,mdsmap=1531151339,mgr=1292546760,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=2364737929,mon_config_key=1784637070,monmap=530371891,osd_metadata=2735459733,osd_pg_creating=4181675810}) 2024-09-18T14:27:50.802 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1} crc {logm=3360953179,mdsmap=1531151339,mgr=3617533101,mgr_command_descs=47663135,mgr_metadata=2046822578,mgrstat=2039505758,mon_config_key=1784637070,monmap=530371891,osd_metadata=2265752427,osd_pg_creating=4181675810}) 2024-09-18T14:27:50.803 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.803 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=3178633393,osdmap=3917215152}) 2024-09-18T14:27:50.803 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=3178633393,osdmap=1421359982}) 2024-09-18T14:27:50.803 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.803 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=3178633393,osdmap=3917215152}) 2024-09-18T14:27:50.803 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=3178633393,osdmap=2227159781}) 2024-09-18T14:27:50.804 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.804 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=349085273}) 2024-09-18T14:27:50.804 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4069389920}) 2024-09-18T14:27:50.804 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.804 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=349085273}) 2024-09-18T14:27:50.804 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.797+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1477013299}) 2024-09-18T14:27:50.806 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.806 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3121266163}) 2024-09-18T14:27:50.806 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3178911066}) 2024-09-18T14:27:50.806 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.806 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3121266163}) 2024-09-18T14:27:50.806 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1781286951}) 2024-09-18T14:27:50.807 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.807 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=1879699159}) 2024-09-18T14:27:50.807 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=11} crc {osdmap=3346393757}) 2024-09-18T14:27:50.807 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T14:27:50.807 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=1879699159}) 2024-09-18T14:27:50.807 INFO:tasks.ceph.mon.a.smithi104.stderr:2024-09-18T14:27:50.801+0000 7fde2914e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=13} crc {osdmap=1344587699}) 2024-09-18T14:27:51.785 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T14:27:51.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T14:27:51.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient: get_monmap_and_config 2024-09-18T14:27:51.894 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient: build_initial_monmap 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient: monmap: 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:51.893124+0000 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:51.893124+0000 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:51.895 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 1 Processor -- start 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 1 -- start start 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient: init 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9400097a50 0x7f9400097e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient(hunting): picked mon.noname-a con 0x7f9400097a50 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f940009aac0 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f940013b4a0 0x7f940013d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient(hunting): picked mon.noname-c con 0x7f940013b4a0 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:51.898 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940009d7f0 con 0x7f940009aac0 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940009da30 con 0x7f9400097a50 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9400096a60 con 0x7f940013b4a0 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9408690700 10 monclient(hunting): authenticate will time out at 1192.882765s 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9400097a50 0x7f9400097e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9400097a50 0x7f9400097e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35592/0 (socket says 172.21.15.104:35592) 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 1 -- 172.21.15.104:0/3002702718 learned_addr learned my addr 172.21.15.104:0/3002702718 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 10 monclient(hunting): get_auth_request con 0x7f9400097a50 auth_method 0 2024-09-18T14:27:51.899 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f940642c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 1 --2- 172.21.15.104:0/3002702718 >> v2:172.21.15.174:3300/0 conn(0x7f940013b4a0 0x7f940013d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): get_auth_request con 0x7f940013b4a0 auth_method 0 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:51.900 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 10 monclient(hunting): handle_auth_done global_id 5821 payload 274 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.889+0000 7f9406c2d700 1 -- 172.21.15.104:0/3002702718 >> v2:172.21.15.104:3300/0 conn(0x7f9400097a50 msgr2=0x7f9400097e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 1 --2- 172.21.15.104:0/3002702718 >> v2:172.21.15.104:3300/0 conn(0x7f9400097a50 0x7f9400097e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 1 -- 172.21.15.104:0/3002702718 >> v1:172.21.15.104:6789/0 conn(0x7f940009aac0 legacy=0x7f940009aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 10 monclient: _finish_hunting 0 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 1 monclient: found mon.noname-c 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:27:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 1 -- 172.21.15.104:0/3002702718 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940009b5d0 con 0x7f940013b4a0 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 1 -- 172.21.15.104:0/3002702718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 msgr2=0x7f940013d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9406c2d700 1 --2- 172.21.15.104:0/3002702718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 0x7f940013d880 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9400096e00 tx=0x7f93fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f579d71d4e5014f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 1 -- 172.21.15.104:0/3002702718 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc011840 con 0x7f940013b4a0 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: dump: 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:51.903 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: _finish_auth 0 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9408690700 5 monclient: authenticate success, global_id 5821 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 1 -- 172.21.15.104:0/3002702718 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc0119a0 con 0x7f940013b4a0 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9408690700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9408690700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f9408690700 4 set_mon_vals no callback set 2024-09-18T14:27:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 1 -- 172.21.15.104:0/3002702718 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc011d40 con 0x7f940013b4a0 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.893+0000 7f940542a700 10 monclient: dump: 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient: shutdown 2024-09-18T14:27:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- 172.21.15.104:0/3002702718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 msgr2=0x7f940013d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 --2- 172.21.15.104:0/3002702718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 0x7f940013d880 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9400096e00 tx=0x7f93fc0091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- 172.21.15.104:0/3002702718 shutdown_connections 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 --2- 172.21.15.104:0/3002702718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 0x7f940013d880 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 --2- 172.21.15.104:0/3002702718 >> v2:172.21.15.104:3300/0 conn(0x7f9400097a50 0x7f9400097e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- 172.21.15.104:0/3002702718 >> 172.21.15.104:0/3002702718 conn(0x7f94000a1290 msgr2=0x7f94000a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- 172.21.15.104:0/3002702718 shutdown_connections 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- 172.21.15.104:0/3002702718 wait complete. 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient: build_initial_monmap 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient: monmap: 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 Processor -- start 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- start start 2024-09-18T14:27:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient: init 2024-09-18T14:27:51.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:51.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:27:51.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 0x7f940006e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:51.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): picked mon.noname-c con 0x7f9400097a50 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:51.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f940009aac0 0x7f9400145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): picked mon.noname-a con 0x7f940009aac0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 0x7f9400145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): picked mon.noname-b con 0x7f940013b4a0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940006c300 con 0x7f940009aac0 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940006bcb0 con 0x7f940013b4a0 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940006b9d0 con 0x7f9400097a50 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9408690700 10 monclient(hunting): authenticate will time out at 1192.890606s 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 0x7f940006e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:51.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f940009aac0 0x7f9400145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 0x7f940006e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42878/0 (socket says 172.21.15.104:42878) 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 -- 172.21.15.104:0/675589794 learned_addr learned my addr 172.21.15.104:0/675589794 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): get_auth_request con 0x7f9400097a50 auth_method 0 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 10 monclient(hunting): get_auth_request con 0x7f940009aac0 auth_method 0 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 0x7f9400145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:51.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 10 monclient(hunting): get_auth_request con 0x7f940013b4a0 auth_method 0 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9406c2d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient(hunting): handle_auth_done global_id 34118 payload 824 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 -- 172.21.15.104:0/675589794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 msgr2=0x7f9400145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 0x7f9400145980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 -- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f940009aac0 msgr2=0x7f9400145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f940009aac0 0x7f9400145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient: _finish_hunting 0 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 monclient: found mon.noname-c 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f940642c700 1 -- 172.21.15.104:0/675589794 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc009420 con 0x7f9400097a50 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.897+0000 7f9405c2b700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f940009aac0 0x7f9400145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f940642c700 1 -- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 msgr2=0x7f940006e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f940642c700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 0x7f940006e7b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009f00 tx=0x7f93f800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 1 -- 172.21.15.104:0/675589794 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8009c30 con 0x7f9400097a50 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: dump: 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f9408690700 5 monclient: authenticate success, global_id 34118 2024-09-18T14:27:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f9408690700 10 monclient: _renew_subs 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 1 -- 172.21.15.104:0/675589794 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f800f040 con 0x7f9400097a50 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f9408690700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940006bf90 con 0x7f9400097a50 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 1 -- 172.21.15.104:0/675589794 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f800cd40 con 0x7f9400097a50 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f7fff700 4 set_mon_vals no callback set 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 10 monclient: dump: 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f9408690700 10 monclient: _renew_subs 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f9408690700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940013af80 con 0x7f9400097a50 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.901+0000 7f93f67fc700 1 -- 172.21.15.104:0/675589794 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f93f8014650 con 0x7f9400097a50 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f93f67fc700 1 --2- 172.21.15.104:0/675589794 >> v2:172.21.15.174:6812/17603 conn(0x7f93e4035a80 0x7f93e4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f9408690700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f9408690700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f9408690700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f93f67fc700 1 -- 172.21.15.104:0/675589794 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f93f8048510 con 0x7f9400097a50 2024-09-18T14:27:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9400064b30 con 0x7f9400097a50 2024-09-18T14:27:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f9405c2b700 1 --2- 172.21.15.104:0/675589794 >> v2:172.21.15.174:6812/17603 conn(0x7f93e4035a80 0x7f93e4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f9405c2b700 10 monclient: get_auth_request con 0x7f93e4035a80 auth_method 0 2024-09-18T14:27:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.905+0000 7f9405c2b700 1 --2- 172.21.15.104:0/675589794 >> v2:172.21.15.174:6812/17603 conn(0x7f93e4035a80 0x7f93e4037f30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f940013d860 tx=0x7f93f0007ea0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:51.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.913+0000 7f93f67fc700 1 -- 172.21.15.104:0/675589794 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f93f801d400 con 0x7f9400097a50 2024-09-18T14:27:51.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.913+0000 7f93f67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:51.920 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:51.913+0000 7f93f67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:52.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.141+0000 7f9408690700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T14:27:52.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.141+0000 7f9408690700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:52.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.141+0000 7f9408690700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.141+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9400068ce0 con 0x7f9400097a50 2024-09-18T14:27:52.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f93f67fc700 1 -- 172.21.15.104:0/675589794 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f93f802a620 con 0x7f9400097a50 2024-09-18T14:27:52.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f93f67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T14:27:52.149 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f93f67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:52.149 INFO:teuthology.orchestra.run.smithi104.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","modified":"2024-09-18T14:21:05.806012Z","created":"2024-09-18T14:21:05.806012Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3301","nonce":0},{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 >> v2:172.21.15.174:6812/17603 conn(0x7f93e4035a80 msgr2=0x7f93e4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 --2- 172.21.15.104:0/675589794 >> v2:172.21.15.174:6812/17603 conn(0x7f93e4035a80 0x7f93e4037f30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f940013d860 tx=0x7f93f0007ea0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 10 monclient: shutdown 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 msgr2=0x7f940006e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 0x7f940006e7b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009f00 tx=0x7f93f800b040 comp rx=0 tx=0).stop 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 shutdown_connections 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 --2- 172.21.15.104:0/675589794 >> v2:172.21.15.174:6812/17603 conn(0x7f93e4035a80 0x7f93e4037f30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f940013b4a0 0x7f9400145980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f940009aac0 0x7f9400145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 --2- 172.21.15.104:0/675589794 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9400097a50 0x7f940006e7b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.152 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 >> 172.21.15.104:0/675589794 conn(0x7f94000a1290 msgr2=0x7f9400098aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:52.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 shutdown_connections 2024-09-18T14:27:52.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.145+0000 7f9408690700 1 -- 172.21.15.104:0/675589794 wait complete. 2024-09-18T14:27:52.163 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T14:27:52.163 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T14:27:52.172 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-18T14:27:52.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient: get_monmap_and_config 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient: build_initial_monmap 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient: monmap: 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:52.273663+0000 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:52.273663+0000 2024-09-18T14:27:52.275 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:52.276 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.276 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:52.276 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:52.276 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:52.276 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:52.276 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:52.276 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 1 Processor -- start 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 1 -- start start 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient: init 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c0a5d10 0x7f8e7c132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c0a5d10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e7c09f160 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e7c09f530 0x7f8e7c0a0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c09f530 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c09d950 con 0x7f8e7c09f160 2024-09-18T14:27:52.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7c0a60e0 con 0x7f8e7c09f530 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0a1f10 con 0x7f8e7c0a5d10 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e80fd5700 10 monclient(hunting): authenticate will time out at 1193.263655s 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e7c09f530 0x7f8e7c0a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e7c09f530 0x7f8e7c0a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35610/0 (socket says 172.21.15.104:35610) 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7ad9d700 1 -- 172.21.15.104:0/3798456236 learned_addr learned my addr 172.21.15.104:0/3798456236 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7ad9d700 10 monclient(hunting): get_auth_request con 0x7f8e7c09f530 auth_method 0 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e7a59c700 1 --2- 172.21.15.104:0/3798456236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c0a5d10 0x7f8e7c132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.269+0000 7f8e79d9b700 1 --1- 172.21.15.104:0/3798456236 >> v1:172.21.15.104:6789/0 conn(0x7f8e7c09f160 0x7f8e7c09fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34806/0 (socket says 172.21.15.104:34806) 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): get_auth_request con 0x7f8e7c0a5d10 auth_method 0 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_done global_id 34121 payload 274 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 1 -- 172.21.15.104:0/3798456236 >> v2:172.21.15.104:3300/0 conn(0x7f8e7c09f530 msgr2=0x7f8e7c0a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:52.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 1 --2- 172.21.15.104:0/3798456236 >> v2:172.21.15.104:3300/0 conn(0x7f8e7c09f530 0x7f8e7c0a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 1 -- 172.21.15.104:0/3798456236 >> v1:172.21.15.104:6789/0 conn(0x7f8e7c09f160 legacy=0x7f8e7c09fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:52.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient: _finish_hunting 0 2024-09-18T14:27:52.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 1 monclient: found mon.noname-b 2024-09-18T14:27:52.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:52.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 1 -- 172.21.15.104:0/3798456236 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c09cae0 con 0x7f8e7c0a5d10 2024-09-18T14:27:52.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 1 -- 172.21.15.104:0/3798456236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c0a5d10 msgr2=0x7f8e7c132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:52.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7a59c700 1 --2- 172.21.15.104:0/3798456236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c0a5d10 0x7f8e7c132290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e700098e0 tx=0x7f8e700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab4fda28e1bac7c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:52.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 1 -- 172.21.15.104:0/3798456236 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000b850 con 0x7f8e7c0a5d10 2024-09-18T14:27:52.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:52.288 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: dump: 2024-09-18T14:27:52.288 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: _finish_auth 0 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e80fd5700 5 monclient: authenticate success, global_id 34121 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 1 -- 172.21.15.104:0/3798456236 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7000b9b0 con 0x7f8e7c0a5d10 2024-09-18T14:27:52.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e80fd5700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e80fd5700 4 set_mon_vals no callback set 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 1 -- 172.21.15.104:0/3798456236 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000bd50 con 0x7f8e7c0a5d10 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.273+0000 7f8e7959a700 10 monclient: dump: 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.290 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient: shutdown 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- 172.21.15.104:0/3798456236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c0a5d10 msgr2=0x7f8e7c132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/3798456236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c0a5d10 0x7f8e7c132290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e700098e0 tx=0x7f8e700091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- 172.21.15.104:0/3798456236 shutdown_connections 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/3798456236 >> v2:172.21.15.104:3300/0 conn(0x7f8e7c09f530 0x7f8e7c0a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/3798456236 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c0a5d10 0x7f8e7c132290 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- 172.21.15.104:0/3798456236 >> 172.21.15.104:0/3798456236 conn(0x7f8e7c0695f0 msgr2=0x7f8e7c0a5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- 172.21.15.104:0/3798456236 shutdown_connections 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- 172.21.15.104:0/3798456236 wait complete. 2024-09-18T14:27:52.291 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient: build_initial_monmap 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient: monmap: 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 Processor -- start 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- start start 2024-09-18T14:27:52.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient: init 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 0x7f8e7c09be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c09f160 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e7c09f530 0x7f8e7c12edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c09f530 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e7c0a5d10 0x7f8e7c12f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c0a5d10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c08be70 con 0x7f8e7c09f530 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c131340 con 0x7f8e7c0a5d10 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c09cd10 con 0x7f8e7c09f160 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e80fd5700 10 monclient(hunting): authenticate will time out at 1193.271064s 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 0x7f8e7c09be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e7c09f530 0x7f8e7c12edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 0x7f8e7c09be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42896/0 (socket says 172.21.15.104:42896) 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e7c09f530 0x7f8e7c12edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35618/0 (socket says 172.21.15.104:35618) 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 -- 172.21.15.104:0/4250405524 learned_addr learned my addr 172.21.15.104:0/4250405524 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): get_auth_request con 0x7f8e7c09f160 auth_method 0 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 10 monclient(hunting): get_auth_request con 0x7f8e7c09f530 auth_method 0 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.294 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7ad9d700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e7c0a5d10 0x7f8e7c12f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7ad9d700 10 monclient(hunting): get_auth_request con 0x7f8e7c0a5d10 auth_method 0 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient(hunting): handle_auth_done global_id 34124 payload 824 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 -- 172.21.15.104:0/4250405524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e7c0a5d10 msgr2=0x7f8e7c12f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e7c0a5d10 0x7f8e7c12f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 -- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e7c09f530 msgr2=0x7f8e7c12edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e7c09f530 0x7f8e7c12edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient: _finish_hunting 0 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 monclient: found mon.noname-c 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e7a59c700 1 -- 172.21.15.104:0/4250405524 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70009420 con 0x7f8e7c09f160 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.277+0000 7f8e79d9b700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e7c09f530 0x7f8e7c12edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e7a59c700 1 -- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 msgr2=0x7f8e7c09be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e7a59c700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 0x7f8e7c09be80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e700030c0 tx=0x7f8e70016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 1 -- 172.21.15.104:0/4250405524 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70011bb0 con 0x7f8e7c09f160 2024-09-18T14:27:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: dump: 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:52.297 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 1 -- 172.21.15.104:0/4250405524 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e70020070 con 0x7f8e7c09f160 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 5 monclient: authenticate success, global_id 34124 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 10 monclient: _renew_subs 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7c0a3d70 con 0x7f8e7c09f160 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 1 -- 172.21.15.104:0/4250405524 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7001b450 con 0x7f8e7c09f160 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e6a7fc700 10 monclient: dump: 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e78d99700 4 set_mon_vals no callback set 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 10 monclient: _renew_subs 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7c130940 con 0x7f8e7c09f160 2024-09-18T14:27:52.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:52.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:52.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.281+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7c08c2e0 con 0x7f8e7c09f160 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.285+0000 7f8e6a7fc700 1 -- 172.21.15.104:0/4250405524 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f8e70022030 con 0x7f8e7c09f160 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.285+0000 7f8e6a7fc700 1 --2- 172.21.15.104:0/4250405524 >> v2:172.21.15.174:6812/17603 conn(0x7f8e5c035ad0 0x7f8e5c037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.285+0000 7f8e6a7fc700 1 -- 172.21.15.104:0/4250405524 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8e70049a60 con 0x7f8e7c09f160 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.285+0000 7f8e79d9b700 1 --2- 172.21.15.104:0/4250405524 >> v2:172.21.15.174:6812/17603 conn(0x7f8e5c035ad0 0x7f8e5c037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.285+0000 7f8e79d9b700 10 monclient: get_auth_request con 0x7f8e5c035ad0 auth_method 0 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.285+0000 7f8e79d9b700 1 --2- 172.21.15.104:0/4250405524 >> v2:172.21.15.174:6812/17603 conn(0x7f8e5c035ad0 0x7f8e5c037f80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009ed0 tx=0x7f8e6c009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.293+0000 7f8e6a7fc700 1 -- 172.21.15.104:0/4250405524 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8e7001b880 con 0x7f8e7c09f160 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.293+0000 7f8e6a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.293+0000 7f8e6a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:52.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.461+0000 7f8e80fd5700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:27:52.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.461+0000 7f8e80fd5700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:27:52.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.461+0000 7f8e80fd5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.461+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e7c08c2e0 con 0x7f8e7c09f160 2024-09-18T14:27:52.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.461+0000 7f8e6a7fc700 1 -- 172.21.15.104:0/4250405524 <== mon.2 v2:172.21.15.104:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v327) v1 ==== 74+0+10033 (secure 0 0 0) 0x7f8e7001a3d0 con 0x7f8e7c09f160 2024-09-18T14:27:52.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.461+0000 7f8e6a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:27:52.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.461+0000 7f8e6a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:52.468 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:27:52.468 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":327,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:27:37.789105+0000","last_up_change":"2024-09-18T14:21:25.123789+0000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T14:21:26.474399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T14:21:30.433492+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":17763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":17763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":17763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":17763}]},"public_addr":"172.21.15.104:6808/17763","cluster_addr":"172.21.15.104:6809/17763","heartbeat_back_addr":"172.21.15.104:6811/17763","heartbeat_front_addr":"172.21.15.104:6810/17763","state":["exists","up"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":17761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":17761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":17761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":17761}]},"public_addr":"172.21.15.104:6804/17761","cluster_addr":"172.21.15.104:6805/17761","heartbeat_back_addr":"172.21.15.104:6807/17761","heartbeat_front_addr":"172.21.15.104:6806/17761","state":["exists","up"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":17762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":17762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":17762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":17762}]},"public_addr":"172.21.15.104:6800/17762","cluster_addr":"172.21.15.104:6801/17762","heartbeat_back_addr":"172.21.15.104:6803/17762","heartbeat_front_addr":"172.21.15.104:6802/17762","state":["exists","up"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":17845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":17845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":17845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":17845}]},"public_addr":"172.21.15.174:6808/17845","cluster_addr":"172.21.15.174:6809/17845","heartbeat_back_addr":"172.21.15.174:6811/17845","heartbeat_front_addr":"172.21.15.174:6810/17845","state":["exists","up"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":17847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6801","nonce":17847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":17847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":17847}]},"public_addr":"172.21.15.174:6800/17847","cluster_addr":"172.21.15.174:6801/17847","heartbeat_back_addr":"172.21.15.174:6803/17847","heartbeat_front_addr":"172.21.15.174:6802/17847","state":["exists","up"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":17846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":17846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":17846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":17846}]},"public_addr":"172.21.15.174:6804/17846","cluster_addr":"172.21.15.174:6805/17846","heartbeat_back_addr":"172.21.15.174:6807/17846","heartbeat_front_addr":"172.21.15.174:6806/17846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.237413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.242543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.214389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.213881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.183415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.230911+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:27:52.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 >> v2:172.21.15.174:6812/17603 conn(0x7f8e5c035ad0 msgr2=0x7f8e5c037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/4250405524 >> v2:172.21.15.174:6812/17603 conn(0x7f8e5c035ad0 0x7f8e5c037f80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009ed0 tx=0x7f8e6c009060 comp rx=0 tx=0).stop 2024-09-18T14:27:52.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 10 monclient: shutdown 2024-09-18T14:27:52.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:52.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 msgr2=0x7f8e7c09be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 0x7f8e7c09be80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e700030c0 tx=0x7f8e70016000 comp rx=0 tx=0).stop 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 shutdown_connections 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/4250405524 >> v2:172.21.15.174:6812/17603 conn(0x7f8e5c035ad0 0x7f8e5c037f80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e7c0a5d10 0x7f8e7c12f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e7c09f530 0x7f8e7c12edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 --2- 172.21.15.104:0/4250405524 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f8e7c09f160 0x7f8e7c09be80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 >> 172.21.15.104:0/4250405524 conn(0x7f8e7c0695f0 msgr2=0x7f8e7c12a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 shutdown_connections 2024-09-18T14:27:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.465+0000 7f8e80fd5700 1 -- 172.21.15.104:0/4250405524 wait complete. 2024-09-18T14:27:52.482 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T14:27:52.482 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient: get_monmap_and_config 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient: build_initial_monmap 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient: monmap: 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:52.586250+0000 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:52.586250+0000 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:52.588 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:52.589 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:52.589 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:52.589 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 1 Processor -- start 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 1 -- start start 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient: init 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab4009a5b0 0x7fab4009ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient(hunting): picked mon.noname-b con 0x7fab4009a5b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab4012f980 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient(hunting): picked mon.noname-a con 0x7fab40130460 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40097d10 con 0x7fab4012f980 2024-09-18T14:27:52.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab40098470 con 0x7fab40130460 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab40099650 con 0x7fab4009a5b0 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.581+0000 7fab45dd8700 10 monclient(hunting): authenticate will time out at 1193.575810s 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab4009a5b0 0x7fab4009ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35630/0 (socket says 172.21.15.104:35630) 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 -- 172.21.15.104:0/1366962966 learned_addr learned my addr 172.21.15.104:0/1366962966 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): get_auth_request con 0x7fab40130460 auth_method 0 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 10 monclient(hunting): get_auth_request con 0x7fab4009a5b0 auth_method 0 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab44dd6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient(hunting): handle_auth_done global_id 15177 payload 274 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 -- 172.21.15.104:0/1366962966 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab4009a5b0 msgr2=0x7fab4009ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 --2- 172.21.15.104:0/1366962966 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab4009a5b0 0x7fab4009ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 -- 172.21.15.104:0/1366962966 >> v1:172.21.15.174:6789/0 conn(0x7fab4012f980 legacy=0x7fab4012fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient: _finish_hunting 0 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 monclient: found mon.noname-a 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 -- 172.21.15.104:0/1366962966 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40099930 con 0x7fab40130460 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 -- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 msgr2=0x7fab40136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 -- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 msgr2=0x7fab40136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fab300099b0 tx=0x7fab30009250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T14:27:52.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.585+0000 7fab455d7700 1 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab455d7700 1 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab455d7700 10 monclient: get_auth_request con 0x7fab40130460 auth_method 0 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab455d7700 0 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab455d7700 1 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient: ms_handle_reset current mon v2:172.21.15.104:3300/0 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 1 -- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 msgr2=0x7fab40136ed0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 1 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.104:3300/0 conn(0x7fab40130460 0x7fab40136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient: _add_conns ranks=[0,4] 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab20003550 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 1 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.174:3300/0 conn(0x7fab200074d0 0x7fab20009980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient(hunting): picked mon.noname-c con 0x7fab200074d0 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 1 -- 172.21.15.104:0/1366962966 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2000a290 con 0x7fab20003550 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 1 -- 172.21.15.104:0/1366962966 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2000b450 con 0x7fab200074d0 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3f7fe700 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.104:3300/0 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 1 --2- 172.21.15.104:0/1366962966 >> v2:172.21.15.174:3300/0 conn(0x7fab200074d0 0x7fab20009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): get_auth_request con 0x7fab200074d0 auth_method 0 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.796 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient(hunting): handle_auth_done global_id 15177 payload 293 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 1 -- 172.21.15.104:0/1366962966 >> v1:172.21.15.104:6789/0 conn(0x7fab20003550 legacy=0x7fab200059c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient: _finish_hunting 0 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 1 monclient: found mon.noname-c 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.789+0000 7fab3ffff700 1 -- 172.21.15.104:0/1366962966 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2000b6d0 con 0x7fab200074d0 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3ffff700 1 -- 172.21.15.104:0/1366962966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 msgr2=0x7fab20009980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:52.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3ffff700 1 --2- 172.21.15.104:0/1366962966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 0x7fab20009980 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fab28002e50 tx=0x7fab28002530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=742488e7c4a00a71 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 1 -- 172.21.15.104:0/1366962966 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28016070 con 0x7fab200074d0 2024-09-18T14:27:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: dump: 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: _finish_auth 0 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 5 monclient: authenticate success, global_id 15177 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 1 -- 172.21.15.104:0/1366962966 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab28006ba0 con 0x7fab200074d0 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 4 set_mon_vals no callback set 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 1 -- 172.21.15.104:0/1366962966 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28002840 con 0x7fab200074d0 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab3f7fe700 10 monclient: dump: 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.800 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.801 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 10 monclient: shutdown 2024-09-18T14:27:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 1 -- 172.21.15.104:0/1366962966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 msgr2=0x7fab20009980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1366962966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 0x7fab20009980 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fab28002e50 tx=0x7fab28002530 comp rx=0 tx=0).stop 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab455d7700 1 -- 172.21.15.104:0/1366962966 reap_dead start 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 1 -- 172.21.15.104:0/1366962966 shutdown_connections 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1366962966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 0x7fab20009980 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 1 -- 172.21.15.104:0/1366962966 >> 172.21.15.104:0/1366962966 conn(0x7fab400a12c0 msgr2=0x7fab400a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 1 -- 172.21.15.104:0/1366962966 shutdown_connections 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.793+0000 7fab45dd8700 1 -- 172.21.15.104:0/1366962966 wait complete. 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient: build_initial_monmap 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient: monmap: 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:52.805 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 Processor -- start 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 -- start start 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient: init 2024-09-18T14:27:52.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 0x7fab40136370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): picked mon.noname-b con 0x7fab200074d0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab40131f60 0x7fab40132350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): picked mon.noname-a con 0x7fab40131f60 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab40132890 0x7fab40145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): picked mon.noname-c con 0x7fab40132890 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4012d790 con 0x7fab40131f60 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab40135c20 con 0x7fab200074d0 2024-09-18T14:27:52.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab400989a0 con 0x7fab40132890 2024-09-18T14:27:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab45dd8700 10 monclient(hunting): authenticate will time out at 1193.789696s 2024-09-18T14:27:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab40131f60 0x7fab40132350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab40132890 0x7fab40145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab40131f60 0x7fab40132350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35654/0 (socket says 172.21.15.104:35654) 2024-09-18T14:27:52.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 1 -- 172.21.15.104:0/1807961356 learned_addr learned my addr 172.21.15.104:0/1807961356 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 10 monclient(hunting): get_auth_request con 0x7fab40131f60 auth_method 0 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): get_auth_request con 0x7fab40132890 auth_method 0 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 0x7fab40136370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 10 monclient(hunting): get_auth_request con 0x7fab200074d0 auth_method 0 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:52.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab44dd6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient(hunting): handle_auth_done global_id 34130 payload 824 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 1 -- 172.21.15.104:0/1807961356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 msgr2=0x7fab40136370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 0x7fab40136370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 1 -- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab40131f60 msgr2=0x7fab40132350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab40131f60 0x7fab40132350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient: _finish_hunting 0 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 1 monclient: found mon.noname-c 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab455d7700 1 -- 172.21.15.104:0/1807961356 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4009cdb0 con 0x7fab40132890 2024-09-18T14:27:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.797+0000 7fab3ffff700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab40131f60 0x7fab40132350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:52.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab455d7700 1 -- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab40132890 msgr2=0x7fab40145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab455d7700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab40132890 0x7fab40145730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab400a0e80 tx=0x7fab30009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 1 -- 172.21.15.104:0/1807961356 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30011e60 con 0x7fab40132890 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: dump: 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.814 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: _finish_auth 0 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 1 -- 172.21.15.104:0/1807961356 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30009ea0 con 0x7fab40132890 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab45dd8700 5 monclient: authenticate success, global_id 34130 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab45dd8700 10 monclient: _renew_subs 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab45dd8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab401333d0 con 0x7fab40132890 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 1 -- 172.21.15.104:0/1807961356 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3001c650 con 0x7fab40132890 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3cff9700 10 monclient: dump: 2024-09-18T14:27:52.815 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab45dd8700 10 monclient: _renew_subs 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab3effd700 4 set_mon_vals no callback set 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab45dd8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.816 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.801+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab40133aa0 con 0x7fab40132890 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab3cff9700 1 -- 172.21.15.104:0/1807961356 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fab3001b3a0 con 0x7fab40132890 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab45dd8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab45dd8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab45dd8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab400633b0 con 0x7fab40132890 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab3cff9700 1 --2- 172.21.15.104:0/1807961356 >> v2:172.21.15.174:6812/17603 conn(0x7fab24035a80 0x7fab24037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab3cff9700 1 -- 172.21.15.104:0/1807961356 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fab30000e40 con 0x7fab40132890 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab44dd6700 1 --2- 172.21.15.104:0/1807961356 >> v2:172.21.15.174:6812/17603 conn(0x7fab24035a80 0x7fab24037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.805+0000 7fab44dd6700 10 monclient: get_auth_request con 0x7fab24035a80 auth_method 0 2024-09-18T14:27:52.817 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.809+0000 7fab44dd6700 1 --2- 172.21.15.104:0/1807961356 >> v2:172.21.15.174:6812/17603 conn(0x7fab24035a80 0x7fab24037f30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fab38009170 tx=0x7fab380091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:52.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.809+0000 7fab3cff9700 1 -- 172.21.15.104:0/1807961356 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fab3000f2e0 con 0x7fab40132890 2024-09-18T14:27:52.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.809+0000 7fab3cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:52.818 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.809+0000 7fab3cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.973+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab40068a50 con 0x7fab24035a80 2024-09-18T14:27:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab3cff9700 1 -- 172.21.15.104:0/1807961356 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51742 (secure 0 0 0) 0x7fab40068a50 con 0x7fab24035a80 2024-09-18T14:27:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:27:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 >> v2:172.21.15.174:6812/17603 conn(0x7fab24035a80 msgr2=0x7fab24037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1807961356 >> v2:172.21.15.174:6812/17603 conn(0x7fab24035a80 0x7fab24037f30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fab38009170 tx=0x7fab380091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 10 monclient: shutdown 2024-09-18T14:27:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab40132890 msgr2=0x7fab40145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab40132890 0x7fab40145730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab400a0e80 tx=0x7fab30009250 comp rx=0 tx=0).stop 2024-09-18T14:27:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 shutdown_connections 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1807961356 >> v2:172.21.15.174:6812/17603 conn(0x7fab24035a80 0x7fab24037f30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fab40132890 0x7fab40145730 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab40131f60 0x7fab40132350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 --2- 172.21.15.104:0/1807961356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab200074d0 0x7fab40136370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 >> 172.21.15.104:0/1807961356 conn(0x7fab400a12c0 msgr2=0x7fab4012e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 shutdown_connections 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:52.981+0000 7fab45dd8700 1 -- 172.21.15.104:0/1807961356 wait complete. 2024-09-18T14:27:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:27:53.000 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T14:27:52.516435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4795856,"kb_used_data":557400424,"kb_used_omap":58312,"kb_used_meta":0,"kb_avail":557400424,"statfs":{"total":575688990720,"available":570778034176,"internally_reserved":0,"allocated":570778034176,"data_stored":570778034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59711673,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003101"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403327+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:27:35.403327+0000","last_peered":"2024-09-18T14:27:35.403327+0000","last_clean":"2024-09-18T14:27:35.403327+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:27:35.403327+0000","last_undegraded":"2024-09-18T14:27:35.403327+0000","last_fullsized":"2024-09-18T14:27:35.403327+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.407340+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:27:35.407340+0000","last_peered":"2024-09-18T14:27:35.407340+0000","last_clean":"2024-09-18T14:27:35.407340+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:27:35.407340+0000","last_undegraded":"2024-09-18T14:27:35.407340+0000","last_fullsized":"2024-09-18T14:27:35.407340+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.405428+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:27:35.405428+0000","last_peered":"2024-09-18T14:27:35.405428+0000","last_clean":"2024-09-18T14:27:35.405428+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:27:35.405428+0000","last_undegraded":"2024-09-18T14:27:35.405428+0000","last_fullsized":"2024-09-18T14:27:35.405428+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.401849+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:27:35.401849+0000","last_peered":"2024-09-18T14:27:35.401849+0000","last_clean":"2024-09-18T14:27:35.401849+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:27:35.401849+0000","last_undegraded":"2024-09-18T14:27:35.401849+0000","last_fullsized":"2024-09-18T14:27:35.401849+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":308,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404309+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:27:35.404309+0000","last_peered":"2024-09-18T14:27:35.404309+0000","last_clean":"2024-09-18T14:27:35.404309+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:27:35.404309+0000","last_undegraded":"2024-09-18T14:27:35.404309+0000","last_fullsized":"2024-09-18T14:27:35.404309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":304,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.008862+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:27:35.008862+0000","last_peered":"2024-09-18T14:27:35.008862+0000","last_clean":"2024-09-18T14:27:35.008862+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:27:35.008862+0000","last_undegraded":"2024-09-18T14:27:35.008862+0000","last_fullsized":"2024-09-18T14:27:35.008862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403712+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:27:35.403712+0000","last_peered":"2024-09-18T14:27:35.403712+0000","last_clean":"2024-09-18T14:27:35.403712+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:27:35.403712+0000","last_undegraded":"2024-09-18T14:27:35.403712+0000","last_fullsized":"2024-09-18T14:27:35.403712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":307,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404551+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:27:35.404551+0000","last_peered":"2024-09-18T14:27:35.404551+0000","last_clean":"2024-09-18T14:27:35.404551+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:27:35.404551+0000","last_undegraded":"2024-09-18T14:27:35.404551+0000","last_fullsized":"2024-09-18T14:27:35.404551+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":403,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403858+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:27:35.403858+0000","last_peered":"2024-09-18T14:27:35.403858+0000","last_clean":"2024-09-18T14:27:35.403858+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:27:35.403858+0000","last_undegraded":"2024-09-18T14:27:35.403858+0000","last_fullsized":"2024-09-18T14:27:35.403858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9390,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9615786,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.77400000000000002,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.90200000000000002,"5min":1.204,"15min":1.204},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.55700000000000005,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.746}]},{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47899999999999998},"max":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91400000000000003,"15min":0.89600000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.38500000000000001},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":3.742},"last":1.0549999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.84299999999999997,"15min":0.85899999999999999},"min":{"1min":0.63400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.6240000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.98299999999999998,"15min":0.89500000000000002},"min":{"1min":0.93400000000000005,"5min":0.623,"15min":0.432},"max":{"1min":1.361,"5min":1.927,"15min":1.927},"last":1.1020000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76400000000000001,"15min":0.79500000000000004},"min":{"1min":0.56799999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.2030000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.999,"15min":1.0309999999999999},"min":{"1min":0.74199999999999999,"5min":0.67700000000000005,"15min":0.57499999999999996},"max":{"1min":1.452,"5min":2.0590000000000002,"15min":4.0919999999999996},"last":0.98499999999999999}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76900000000000002,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.944,"5min":1.944,"15min":3.218},"last":0.876},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.92600000000000005,"15min":1.006},"min":{"1min":0.72199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.5779999999999998},"last":0.95299999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799008,"kb_used_data":92900372,"kb_used_omap":9896,"kb_used_meta":0,"kb_avail":92900372,"statfs":{"total":95948165120,"available":95129980928,"internally_reserved":0,"allocated":95129980928,"data_stored":95129980928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10133777,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":1.3260000000000001,"5min":1.988,"15min":1.988},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.91000000000000003,"15min":0.97199999999999998},"min":{"1min":0.72199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.111,"5min":2.012,"15min":2.012},"last":0.78300000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.79300000000000004},"min":{"1min":0.66400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0980000000000001,"5min":2.714,"15min":2.714},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89100000000000001,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93999999999999995,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.65100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.875,"15min":0.91500000000000004},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.083,"5min":1.345,"15min":1.99},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94699999999999995,"15min":1.0109999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.046,"5min":1.361,"15min":2.1240000000000001},"last":0.93799999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.69199999999999995,"15min":0.72799999999999998},"min":{"1min":0.56399999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.222},"last":0.626},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1990000000000001,"15min":1.7769999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73099999999999998,"15min":0.88700000000000001},"min":{"1min":0.56299999999999994,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.012,"5min":1.2070000000000001,"15min":3.0609999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84499999999999997,"15min":0.99099999999999999},"min":{"1min":0.65200000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.075,"5min":1.29,"15min":3.2799999999999998},"last":0.68500000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":9529,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9758155,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81399999999999995,"15min":0.93000000000000005},"min":{"1min":0.56100000000000005,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0109999999999999,"5min":3.8359999999999999,"15min":5.9100000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.98299999999999998},"min":{"1min":0.52500000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0429999999999999,"5min":3.5609999999999999,"15min":6.173},"last":0.59899999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.89600000000000002,"15min":1.034},"min":{"1min":0.53300000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.131,"5min":4.2519999999999998,"15min":5.8789999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.95599999999999996,"15min":0.89400000000000002},"min":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3140000000000001,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":0.755}]},{"osd":2,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90600000000000003,"15min":1.034},"min":{"1min":0.70399999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2250000000000001,"5min":5.9530000000000003,"15min":6.3109999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.95999999999999996,"15min":1.0649999999999999},"min":{"1min":0.71899999999999997,"5min":0.499,"15min":0.499},"max":{"1min":1.0880000000000001,"5min":5.6159999999999997,"15min":6.2350000000000003},"last":0.82699999999999996}]},{"osd":4,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.095,"5min":2.2109999999999999,"15min":3.8420000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79800000000000004,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.99399999999999999,"5min":2.4169999999999998,"15min":5.8410000000000002},"last":0.63900000000000001}]},{"osd":5,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.81599999999999995,"15min":0.93700000000000006},"min":{"1min":0.61199999999999999,"5min":0.53700000000000003,"15min":0.52100000000000002},"max":{"1min":1.167,"5min":1.581,"15min":4.0739999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92500000000000004,"15min":1.01},"min":{"1min":0.628,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.397,"5min":2.6899999999999999,"15min":4.0940000000000003},"last":0.874}]}]},{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799136,"kb_used_data":92900244,"kb_used_omap":9735,"kb_used_meta":0,"kb_avail":92900244,"statfs":{"total":95948165120,"available":95129849856,"internally_reserved":0,"allocated":95129849856,"data_stored":95129849856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9969199,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59399999999999997,"15min":0.56899999999999995},"min":{"1min":0.46700000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.878,"5min":1.849,"15min":1.849},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.68600000000000005,"15min":0.59799999999999998},"min":{"1min":0.52900000000000003,"5min":0.377,"15min":0.376},"max":{"1min":1.375,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78000000000000003,"15min":0.70199999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.206,"5min":4.7640000000000002,"15min":4.7640000000000002},"last":0.627},{"interface":"front","average":{"1min":0.748,"5min":0.84999999999999998,"15min":0.77700000000000002},"min":{"1min":0.56100000000000005,"5min":0.308,"15min":0.308},"max":{"1min":0.92800000000000005,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92900000000000005,"15min":0.89200000000000002},"min":{"1min":0.70499999999999996,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2350000000000001,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.92200000000000004},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.2729999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.93600000000000005}]},{"osd":4,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.91100000000000003,"15min":1.1830000000000001},"min":{"1min":0.69499999999999995,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.2949999999999999,"5min":1.778,"15min":9.0830000000000002},"last":0.996},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95899999999999996,"15min":1.1359999999999999},"min":{"1min":0.75700000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.218,"5min":2.0089999999999999,"15min":7.1399999999999997},"last":0.81599999999999995}]},{"osd":5,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.96099999999999997,"15min":1.1850000000000001},"min":{"1min":0.79400000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.27,"5min":2.0859999999999999,"15min":9.4900000000000002},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.94699999999999995,"15min":1.155},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0369999999999999,"5min":2.2429999999999999,"15min":8.3420000000000005},"last":0.86399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9693,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9926148,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.44700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":1.508,"15min":1.508},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.28599999999999998},"max":{"1min":0.81599999999999995,"5min":1.038,"15min":1.038},"last":0.73499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.251},"max":{"1min":0.92200000000000004,"5min":2.278,"15min":2.278},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74399999999999999,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.70099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.88900000000000001,"15min":1.022},"min":{"1min":0.66000000000000003,"5min":0.41499999999999998,"15min":0.34699999999999998},"max":{"1min":1.0109999999999999,"5min":2.8239999999999998,"15min":5.6699999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88,"5min":0.95099999999999996,"15min":0.84899999999999998},"min":{"1min":0.67900000000000005,"5min":0.53500000000000003,"15min":0.32900000000000001},"max":{"1min":1.196,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.85399999999999998}]},{"osd":4,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88600000000000001,"15min":1.002},"min":{"1min":0.72199999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.079,"5min":1.45,"15min":3.7490000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.92900000000000005,"15min":0.86899999999999999},"min":{"1min":0.65300000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.018,"5min":6.141,"15min":6.141},"last":0.78800000000000003}]},{"osd":5,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.873,"15min":0.98999999999999999},"min":{"1min":0.70899999999999996,"5min":0.46800000000000003,"15min":0.40600000000000003},"max":{"1min":1.111,"5min":1.292,"15min":4.2229999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94199999999999995,"15min":0.91200000000000003},"min":{"1min":0.69899999999999995,"5min":0.46100000000000002,"15min":0.41799999999999998},"max":{"1min":1.198,"5min":1.496,"15min":1.496},"last":0.96499999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640336,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":10067,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10308608,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31},"max":{"1min":0.753,"5min":1.298,"15min":1.298},"last":0.47099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.80600000000000005,"5min":1.48,"15min":1.48},"last":0.505},{"interface":"front","average":{"1min":0.755,"5min":0.71199999999999997,"15min":0.69199999999999995},"min":{"1min":0.51400000000000001,"5min":0.46700000000000003,"15min":0.313},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.872,"15min":0.84499999999999997},"min":{"1min":0.65800000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1759999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.53100000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97499999999999998,"5min":1.212,"15min":1.212},"last":0.82199999999999995}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.97699999999999998},"min":{"1min":0.76900000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2470000000000001,"5min":1.579,"15min":1.9470000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.877,"15min":0.98299999999999998},"min":{"1min":0.627,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":2.476},"last":0.92400000000000004}]},{"osd":5,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.91300000000000003,"15min":0.89000000000000001},"min":{"1min":0.69299999999999995,"5min":0.56399999999999995,"15min":0.56299999999999994},"max":{"1min":1.2989999999999999,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93200000000000005,"15min":0.99399999999999999},"min":{"1min":0.73099999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1539999999999999,"5min":1.3009999999999999,"15min":1.7989999999999999},"last":0.79600000000000004}]}]}],"pool_statfs":[]}} 2024-09-18T14:27:53.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient: get_monmap_and_config 2024-09-18T14:27:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient: build_initial_monmap 2024-09-18T14:27:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient: monmap: 2024-09-18T14:27:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:53.106 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:53.104929+0000 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:53.104929+0000 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:53.107 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 1 Processor -- start 2024-09-18T14:27:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 1 -- start start 2024-09-18T14:27:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient: init 2024-09-18T14:27:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T14:27:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8f0136200 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:53.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8f01365d0 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f013c3a0 0x7fb8f013e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient(hunting): picked mon.noname-b con 0x7fb8f013c3a0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f0095a90 con 0x7fb8f0136200 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f0095cf0 con 0x7fb8f01365d0 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00970b0 con 0x7fb8f013c3a0 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f7437700 10 monclient(hunting): authenticate will time out at 1194.094494s 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6435700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb8f0136200 0x7fb8f009a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34816/0 (socket says 172.21.15.104:34816) 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6435700 1 -- 172.21.15.104:0/547240744 learned_addr learned my addr 172.21.15.104:0/547240744 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 1 --2- 172.21.15.104:0/547240744 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f013c3a0 0x7fb8f013e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): get_auth_request con 0x7fb8f013c3a0 auth_method 0 2024-09-18T14:27:53.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient(hunting): handle_auth_done global_id 34133 payload 274 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 1 -- 172.21.15.104:0/547240744 >> v1:172.21.15.174:6789/0 conn(0x7fb8f01365d0 legacy=0x7fb8f009ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 1 -- 172.21.15.104:0/547240744 >> v1:172.21.15.104:6789/0 conn(0x7fb8f0136200 legacy=0x7fb8f009a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient: _finish_hunting 0 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 1 monclient: found mon.noname-b 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:53.112 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.101+0000 7fb8f6c36700 1 -- 172.21.15.104:0/547240744 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0097390 con 0x7fb8f013c3a0 2024-09-18T14:27:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f6c36700 1 -- 172.21.15.104:0/547240744 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f013c3a0 msgr2=0x7fb8f013e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f6c36700 1 --2- 172.21.15.104:0/547240744 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f013c3a0 0x7fb8f013e780 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009e90 tx=0x7fb8e8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=847bac2994db53ef server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 1 -- 172.21.15.104:0/547240744 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800e070 con 0x7fb8f013c3a0 2024-09-18T14:27:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: dump: 2024-09-18T14:27:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.118 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.119 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: _finish_auth 0 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 1 -- 172.21.15.104:0/547240744 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e8012430 con 0x7fb8f013c3a0 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 1 -- 172.21.15.104:0/547240744 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e80127d0 con 0x7fb8f013c3a0 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 5 monclient: authenticate success, global_id 34133 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 4 set_mon_vals no callback set 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f5433700 10 monclient: dump: 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.121 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 10 monclient: shutdown 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 1 -- 172.21.15.104:0/547240744 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f013c3a0 msgr2=0x7fb8f013e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 1 --2- 172.21.15.104:0/547240744 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f013c3a0 0x7fb8f013e780 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009e90 tx=0x7fb8e8009780 comp rx=0 tx=0).stop 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 1 -- 172.21.15.104:0/547240744 shutdown_connections 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 1 --2- 172.21.15.104:0/547240744 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f013c3a0 0x7fb8f013e780 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 1 -- 172.21.15.104:0/547240744 >> 172.21.15.104:0/547240744 conn(0x7fb8f00a12c0 msgr2=0x7fb8f00a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 1 -- 172.21.15.104:0/547240744 shutdown_connections 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.105+0000 7fb8f7437700 1 -- 172.21.15.104:0/547240744 wait complete. 2024-09-18T14:27:53.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient: build_initial_monmap 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient: monmap: 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 Processor -- start 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 -- start start 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient: init 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:53.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8f0136200 0x7fb8f006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): picked mon.noname-a con 0x7fb8f0136200 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 0x7fb8f01370e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): picked mon.noname-c con 0x7fb8f01365d0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8f013c3a0 0x7fb8f0137620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): picked mon.noname-b con 0x7fb8f013c3a0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f009a5c0 con 0x7fb8f0136200 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f013a130 con 0x7fb8f013c3a0 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0139e50 con 0x7fb8f01365d0 2024-09-18T14:27:53.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f7437700 10 monclient(hunting): authenticate will time out at 1194.101618s 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8f0136200 0x7fb8f006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 0x7fb8f01370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8f0136200 0x7fb8f006f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35660/0 (socket says 172.21.15.104:35660) 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 1 -- 172.21.15.104:0/2431631118 learned_addr learned my addr 172.21.15.104:0/2431631118 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 10 monclient(hunting): get_auth_request con 0x7fb8f0136200 auth_method 0 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 0x7fb8f01370e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42942/0 (socket says 172.21.15.104:42942) 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): get_auth_request con 0x7fb8f01365d0 auth_method 0 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8f013c3a0 0x7fb8f0137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 10 monclient(hunting): get_auth_request con 0x7fb8f013c3a0 auth_method 0 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6c36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient(hunting): handle_auth_done global_id 34136 payload 824 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 -- 172.21.15.104:0/2431631118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8f013c3a0 msgr2=0x7fb8f0137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8f013c3a0 0x7fb8f0137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 -- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8f0136200 msgr2=0x7fb8f006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8f0136200 0x7fb8f006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient: _finish_hunting 0 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 monclient: found mon.noname-c 2024-09-18T14:27:53.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:53.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:53.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f5c34700 1 -- 172.21.15.104:0/2431631118 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e8009a20 con 0x7fb8f01365d0 2024-09-18T14:27:53.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.109+0000 7fb8f6435700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8f0136200 0x7fb8f006f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:53.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f5c34700 1 -- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 msgr2=0x7fb8f01370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:53.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f5c34700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 0x7fb8f01370e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00c490 tx=0x7fb8ec007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 1 -- 172.21.15.104:0/2431631118 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec00ce00 con 0x7fb8f01365d0 2024-09-18T14:27:53.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:53.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: dump: 2024-09-18T14:27:53.130 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 1 -- 172.21.15.104:0/2431631118 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec0117a0 con 0x7fb8f01365d0 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:53.131 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 1 -- 172.21.15.104:0/2431631118 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec011b60 con 0x7fb8f01365d0 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 10 monclient: dump: 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.132 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 5 monclient: authenticate success, global_id 34136 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 10 monclient: _renew_subs 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f4c32700 4 set_mon_vals no callback set 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f013a470 con 0x7fb8f01365d0 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 10 monclient: _renew_subs 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f013b1f0 con 0x7fb8f01365d0 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f0138350 con 0x7fb8f01365d0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 1 -- 172.21.15.104:0/2431631118 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fb8ec01c460 con 0x7fb8f01365d0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 1 --2- 172.21.15.104:0/2431631118 >> v2:172.21.15.174:6812/17603 conn(0x7fb8d4035ad0 0x7fb8d4037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8e67fc700 1 -- 172.21.15.104:0/2431631118 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb8ec047a70 con 0x7fb8f01365d0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f6435700 1 --2- 172.21.15.104:0/2431631118 >> v2:172.21.15.174:6812/17603 conn(0x7fb8d4035ad0 0x7fb8d4037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.113+0000 7fb8f6435700 10 monclient: get_auth_request con 0x7fb8d4035ad0 auth_method 0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.117+0000 7fb8f6435700 1 --2- 172.21.15.104:0/2431631118 >> v2:172.21.15.174:6812/17603 conn(0x7fb8d4035ad0 0x7fb8d4037f80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00052e0 tx=0x7fb8e0004a80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.125+0000 7fb8e67fc700 1 -- 172.21.15.104:0/2431631118 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb8ec0108c0 con 0x7fb8f01365d0 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.125+0000 7fb8e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.125+0000 7fb8e67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:53.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.285+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8f013a2a0 con 0x7fb8d4035ad0 2024-09-18T14:27:53.295 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8e67fc700 1 -- 172.21.15.104:0/2431631118 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51742 (secure 0 0 0) 0x7fb8f013a2a0 con 0x7fb8d4035ad0 2024-09-18T14:27:53.296 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:27:53.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 >> v2:172.21.15.174:6812/17603 conn(0x7fb8d4035ad0 msgr2=0x7fb8d4037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 --2- 172.21.15.104:0/2431631118 >> v2:172.21.15.174:6812/17603 conn(0x7fb8d4035ad0 0x7fb8d4037f80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00052e0 tx=0x7fb8e0004a80 comp rx=0 tx=0).stop 2024-09-18T14:27:53.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 10 monclient: shutdown 2024-09-18T14:27:53.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:53.298 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 msgr2=0x7fb8f01370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 0x7fb8f01370e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00c490 tx=0x7fb8ec007ea0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 shutdown_connections 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 --2- 172.21.15.104:0/2431631118 >> v2:172.21.15.174:6812/17603 conn(0x7fb8d4035ad0 0x7fb8d4037f80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8f013c3a0 0x7fb8f0137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb8f01365d0 0x7fb8f01370e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 --2- 172.21.15.104:0/2431631118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8f0136200 0x7fb8f006f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 >> 172.21.15.104:0/2431631118 conn(0x7fb8f00a12c0 msgr2=0x7fb8f013e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.289+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 shutdown_connections 2024-09-18T14:27:53.299 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.293+0000 7fb8f7437700 1 -- 172.21.15.104:0/2431631118 wait complete. 2024-09-18T14:27:53.300 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:27:53.309 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T14:27:52.516435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4795856,"kb_used_data":557400424,"kb_used_omap":58312,"kb_used_meta":0,"kb_avail":557400424,"statfs":{"total":575688990720,"available":570778034176,"internally_reserved":0,"allocated":570778034176,"data_stored":570778034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59711673,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003101"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403327+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:27:35.403327+0000","last_peered":"2024-09-18T14:27:35.403327+0000","last_clean":"2024-09-18T14:27:35.403327+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:27:35.403327+0000","last_undegraded":"2024-09-18T14:27:35.403327+0000","last_fullsized":"2024-09-18T14:27:35.403327+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.407340+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:27:35.407340+0000","last_peered":"2024-09-18T14:27:35.407340+0000","last_clean":"2024-09-18T14:27:35.407340+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:27:35.407340+0000","last_undegraded":"2024-09-18T14:27:35.407340+0000","last_fullsized":"2024-09-18T14:27:35.407340+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.405428+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:27:35.405428+0000","last_peered":"2024-09-18T14:27:35.405428+0000","last_clean":"2024-09-18T14:27:35.405428+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:27:35.405428+0000","last_undegraded":"2024-09-18T14:27:35.405428+0000","last_fullsized":"2024-09-18T14:27:35.405428+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.401849+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:27:35.401849+0000","last_peered":"2024-09-18T14:27:35.401849+0000","last_clean":"2024-09-18T14:27:35.401849+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:27:35.401849+0000","last_undegraded":"2024-09-18T14:27:35.401849+0000","last_fullsized":"2024-09-18T14:27:35.401849+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":308,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404309+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:27:35.404309+0000","last_peered":"2024-09-18T14:27:35.404309+0000","last_clean":"2024-09-18T14:27:35.404309+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:27:35.404309+0000","last_undegraded":"2024-09-18T14:27:35.404309+0000","last_fullsized":"2024-09-18T14:27:35.404309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":304,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.008862+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:27:35.008862+0000","last_peered":"2024-09-18T14:27:35.008862+0000","last_clean":"2024-09-18T14:27:35.008862+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:27:35.008862+0000","last_undegraded":"2024-09-18T14:27:35.008862+0000","last_fullsized":"2024-09-18T14:27:35.008862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403712+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:27:35.403712+0000","last_peered":"2024-09-18T14:27:35.403712+0000","last_clean":"2024-09-18T14:27:35.403712+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:27:35.403712+0000","last_undegraded":"2024-09-18T14:27:35.403712+0000","last_fullsized":"2024-09-18T14:27:35.403712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":307,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404551+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:27:35.404551+0000","last_peered":"2024-09-18T14:27:35.404551+0000","last_clean":"2024-09-18T14:27:35.404551+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:27:35.404551+0000","last_undegraded":"2024-09-18T14:27:35.404551+0000","last_fullsized":"2024-09-18T14:27:35.404551+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":403,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403858+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:27:35.403858+0000","last_peered":"2024-09-18T14:27:35.403858+0000","last_clean":"2024-09-18T14:27:35.403858+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:27:35.403858+0000","last_undegraded":"2024-09-18T14:27:35.403858+0000","last_fullsized":"2024-09-18T14:27:35.403858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9390,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9615786,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.77400000000000002,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.90200000000000002,"5min":1.204,"15min":1.204},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.55700000000000005,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.746}]},{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47899999999999998},"max":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91400000000000003,"15min":0.89600000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.38500000000000001},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":3.742},"last":1.0549999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.84299999999999997,"15min":0.85899999999999999},"min":{"1min":0.63400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.6240000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.98299999999999998,"15min":0.89500000000000002},"min":{"1min":0.93400000000000005,"5min":0.623,"15min":0.432},"max":{"1min":1.361,"5min":1.927,"15min":1.927},"last":1.1020000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76400000000000001,"15min":0.79500000000000004},"min":{"1min":0.56799999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.2030000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.999,"15min":1.0309999999999999},"min":{"1min":0.74199999999999999,"5min":0.67700000000000005,"15min":0.57499999999999996},"max":{"1min":1.452,"5min":2.0590000000000002,"15min":4.0919999999999996},"last":0.98499999999999999}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76900000000000002,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.944,"5min":1.944,"15min":3.218},"last":0.876},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.92600000000000005,"15min":1.006},"min":{"1min":0.72199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.5779999999999998},"last":0.95299999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799008,"kb_used_data":92900372,"kb_used_omap":9896,"kb_used_meta":0,"kb_avail":92900372,"statfs":{"total":95948165120,"available":95129980928,"internally_reserved":0,"allocated":95129980928,"data_stored":95129980928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10133777,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":1.3260000000000001,"5min":1.988,"15min":1.988},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.91000000000000003,"15min":0.97199999999999998},"min":{"1min":0.72199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.111,"5min":2.012,"15min":2.012},"last":0.78300000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.79300000000000004},"min":{"1min":0.66400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0980000000000001,"5min":2.714,"15min":2.714},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89100000000000001,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93999999999999995,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.65100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.875,"15min":0.91500000000000004},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.083,"5min":1.345,"15min":1.99},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94699999999999995,"15min":1.0109999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.046,"5min":1.361,"15min":2.1240000000000001},"last":0.93799999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.69199999999999995,"15min":0.72799999999999998},"min":{"1min":0.56399999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.222},"last":0.626},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1990000000000001,"15min":1.7769999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73099999999999998,"15min":0.88700000000000001},"min":{"1min":0.56299999999999994,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.012,"5min":1.2070000000000001,"15min":3.0609999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84499999999999997,"15min":0.99099999999999999},"min":{"1min":0.65200000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.075,"5min":1.29,"15min":3.2799999999999998},"last":0.68500000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":9529,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9758155,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81399999999999995,"15min":0.93000000000000005},"min":{"1min":0.56100000000000005,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0109999999999999,"5min":3.8359999999999999,"15min":5.9100000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.98299999999999998},"min":{"1min":0.52500000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0429999999999999,"5min":3.5609999999999999,"15min":6.173},"last":0.59899999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.89600000000000002,"15min":1.034},"min":{"1min":0.53300000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.131,"5min":4.2519999999999998,"15min":5.8789999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.95599999999999996,"15min":0.89400000000000002},"min":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3140000000000001,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":0.755}]},{"osd":2,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90600000000000003,"15min":1.034},"min":{"1min":0.70399999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2250000000000001,"5min":5.9530000000000003,"15min":6.3109999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.95999999999999996,"15min":1.0649999999999999},"min":{"1min":0.71899999999999997,"5min":0.499,"15min":0.499},"max":{"1min":1.0880000000000001,"5min":5.6159999999999997,"15min":6.2350000000000003},"last":0.82699999999999996}]},{"osd":4,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.095,"5min":2.2109999999999999,"15min":3.8420000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79800000000000004,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.99399999999999999,"5min":2.4169999999999998,"15min":5.8410000000000002},"last":0.63900000000000001}]},{"osd":5,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.81599999999999995,"15min":0.93700000000000006},"min":{"1min":0.61199999999999999,"5min":0.53700000000000003,"15min":0.52100000000000002},"max":{"1min":1.167,"5min":1.581,"15min":4.0739999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92500000000000004,"15min":1.01},"min":{"1min":0.628,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.397,"5min":2.6899999999999999,"15min":4.0940000000000003},"last":0.874}]}]},{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799136,"kb_used_data":92900244,"kb_used_omap":9735,"kb_used_meta":0,"kb_avail":92900244,"statfs":{"total":95948165120,"available":95129849856,"internally_reserved":0,"allocated":95129849856,"data_stored":95129849856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9969199,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59399999999999997,"15min":0.56899999999999995},"min":{"1min":0.46700000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.878,"5min":1.849,"15min":1.849},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.68600000000000005,"15min":0.59799999999999998},"min":{"1min":0.52900000000000003,"5min":0.377,"15min":0.376},"max":{"1min":1.375,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78000000000000003,"15min":0.70199999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.206,"5min":4.7640000000000002,"15min":4.7640000000000002},"last":0.627},{"interface":"front","average":{"1min":0.748,"5min":0.84999999999999998,"15min":0.77700000000000002},"min":{"1min":0.56100000000000005,"5min":0.308,"15min":0.308},"max":{"1min":0.92800000000000005,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92900000000000005,"15min":0.89200000000000002},"min":{"1min":0.70499999999999996,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2350000000000001,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.92200000000000004},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.2729999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.93600000000000005}]},{"osd":4,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.91100000000000003,"15min":1.1830000000000001},"min":{"1min":0.69499999999999995,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.2949999999999999,"5min":1.778,"15min":9.0830000000000002},"last":0.996},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95899999999999996,"15min":1.1359999999999999},"min":{"1min":0.75700000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.218,"5min":2.0089999999999999,"15min":7.1399999999999997},"last":0.81599999999999995}]},{"osd":5,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.96099999999999997,"15min":1.1850000000000001},"min":{"1min":0.79400000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.27,"5min":2.0859999999999999,"15min":9.4900000000000002},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.94699999999999995,"15min":1.155},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0369999999999999,"5min":2.2429999999999999,"15min":8.3420000000000005},"last":0.86399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9693,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9926148,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.44700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":1.508,"15min":1.508},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.28599999999999998},"max":{"1min":0.81599999999999995,"5min":1.038,"15min":1.038},"last":0.73499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.251},"max":{"1min":0.92200000000000004,"5min":2.278,"15min":2.278},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74399999999999999,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.70099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.88900000000000001,"15min":1.022},"min":{"1min":0.66000000000000003,"5min":0.41499999999999998,"15min":0.34699999999999998},"max":{"1min":1.0109999999999999,"5min":2.8239999999999998,"15min":5.6699999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88,"5min":0.95099999999999996,"15min":0.84899999999999998},"min":{"1min":0.67900000000000005,"5min":0.53500000000000003,"15min":0.32900000000000001},"max":{"1min":1.196,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.85399999999999998}]},{"osd":4,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88600000000000001,"15min":1.002},"min":{"1min":0.72199999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.079,"5min":1.45,"15min":3.7490000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.92900000000000005,"15min":0.86899999999999999},"min":{"1min":0.65300000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.018,"5min":6.141,"15min":6.141},"last":0.78800000000000003}]},{"osd":5,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.873,"15min":0.98999999999999999},"min":{"1min":0.70899999999999996,"5min":0.46800000000000003,"15min":0.40600000000000003},"max":{"1min":1.111,"5min":1.292,"15min":4.2229999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94199999999999995,"15min":0.91200000000000003},"min":{"1min":0.69899999999999995,"5min":0.46100000000000002,"15min":0.41799999999999998},"max":{"1min":1.198,"5min":1.496,"15min":1.496},"last":0.96499999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640336,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":10067,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10308608,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31},"max":{"1min":0.753,"5min":1.298,"15min":1.298},"last":0.47099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.80600000000000005,"5min":1.48,"15min":1.48},"last":0.505},{"interface":"front","average":{"1min":0.755,"5min":0.71199999999999997,"15min":0.69199999999999995},"min":{"1min":0.51400000000000001,"5min":0.46700000000000003,"15min":0.313},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.872,"15min":0.84499999999999997},"min":{"1min":0.65800000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1759999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.53100000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97499999999999998,"5min":1.212,"15min":1.212},"last":0.82199999999999995}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.97699999999999998},"min":{"1min":0.76900000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2470000000000001,"5min":1.579,"15min":1.9470000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.877,"15min":0.98299999999999998},"min":{"1min":0.627,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":2.476},"last":0.92400000000000004}]},{"osd":5,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.91300000000000003,"15min":0.89000000000000001},"min":{"1min":0.69299999999999995,"5min":0.56399999999999995,"15min":0.56299999999999994},"max":{"1min":1.2989999999999999,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93200000000000005,"15min":0.99399999999999999},"min":{"1min":0.73099999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1539999999999999,"5min":1.3009999999999999,"15min":1.7989999999999999},"last":0.79600000000000004}]}]}],"pool_statfs":[]}} 2024-09-18T14:27:53.311 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T14:27:53.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient: get_monmap_and_config 2024-09-18T14:27:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient: build_initial_monmap 2024-09-18T14:27:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient: monmap: 2024-09-18T14:27:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:53.414440+0000 2024-09-18T14:27:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:53.414440+0000 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:53.417 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 1 Processor -- start 2024-09-18T14:27:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 1 -- start start 2024-09-18T14:27:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient: init 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0334136200 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03341365d0 0x7f033409ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient(hunting): picked mon.noname-a con 0x7f03341365d0 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f033413b390 0x7f033413d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient(hunting): picked mon.noname-b con 0x7f033413b390 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0334095a90 con 0x7f0334136200 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0334095cf0 con 0x7f03341365d0 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03340970b0 con 0x7f033413b390 2024-09-18T14:27:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.409+0000 7f033359e700 10 monclient(hunting): authenticate will time out at 1194.403975s 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0334136200 0x7f033409a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34818/0 (socket says 172.21.15.104:34818) 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033259c700 1 -- 172.21.15.104:0/1610866102 learned_addr learned my addr 172.21.15.104:0/1610866102 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 --2- 172.21.15.104:0/1610866102 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f033413b390 0x7f033413d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 1 --2- 172.21.15.104:0/1610866102 >> v2:172.21.15.104:3300/0 conn(0x7f03341365d0 0x7f033409ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): get_auth_request con 0x7f033413b390 auth_method 0 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 10 monclient(hunting): get_auth_request con 0x7f03341365d0 auth_method 0 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0331d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient(hunting): handle_auth_done global_id 34139 payload 274 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 -- 172.21.15.104:0/1610866102 >> v2:172.21.15.104:3300/0 conn(0x7f03341365d0 msgr2=0x7f033409ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 --2- 172.21.15.104:0/1610866102 >> v2:172.21.15.104:3300/0 conn(0x7f03341365d0 0x7f033409ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 -- 172.21.15.104:0/1610866102 >> v1:172.21.15.104:6789/0 conn(0x7f0334136200 legacy=0x7f033409a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient: _finish_hunting 0 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 monclient: found mon.noname-b 2024-09-18T14:27:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 -- 172.21.15.104:0/1610866102 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334097390 con 0x7f033413b390 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 -- 172.21.15.104:0/1610866102 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f033413b390 msgr2=0x7f033413d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f0332d9d700 1 --2- 172.21.15.104:0/1610866102 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f033413b390 0x7f033413d770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f03240098e0 tx=0x7f03240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19bc6cc85c226782 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 1 -- 172.21.15.104:0/1610866102 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032400b850 con 0x7f033413b390 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: dump: 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: _finish_auth 0 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033359e700 5 monclient: authenticate success, global_id 34139 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 1 -- 172.21.15.104:0/1610866102 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032400b9b0 con 0x7f033413b390 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033359e700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033359e700 4 set_mon_vals no callback set 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 1 -- 172.21.15.104:0/1610866102 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032400bd50 con 0x7f033413b390 2024-09-18T14:27:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.413+0000 7f033159a700 10 monclient: dump: 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.433 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient: shutdown 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- 172.21.15.104:0/1610866102 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f033413b390 msgr2=0x7f033413d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 --2- 172.21.15.104:0/1610866102 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f033413b390 0x7f033413d770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f03240098e0 tx=0x7f03240091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- 172.21.15.104:0/1610866102 shutdown_connections 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 --2- 172.21.15.104:0/1610866102 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f033413b390 0x7f033413d770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 --2- 172.21.15.104:0/1610866102 >> v2:172.21.15.104:3300/0 conn(0x7f03341365d0 0x7f033409ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- 172.21.15.104:0/1610866102 >> 172.21.15.104:0/1610866102 conn(0x7f03340a12c0 msgr2=0x7f03340a16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- 172.21.15.104:0/1610866102 shutdown_connections 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- 172.21.15.104:0/1610866102 wait complete. 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient: build_initial_monmap 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient: monmap: 2024-09-18T14:27:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 Processor -- start 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- start start 2024-09-18T14:27:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient: init 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0334136200 0x7f0334145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): picked mon.noname-c con 0x7f0334136200 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 0x7f0334135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): picked mon.noname-a con 0x7f03341365d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033413b390 0x7f033406ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): picked mon.noname-b con 0x7f033413b390 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03340a06d0 con 0x7f03341365d0 2024-09-18T14:27:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0334146750 con 0x7f033413b390 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0334121610 con 0x7f0334136200 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033359e700 10 monclient(hunting): authenticate will time out at 1194.409669s 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 0x7f0334135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 0x7f0334135220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35672/0 (socket says 172.21.15.104:35672) 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 -- 172.21.15.104:0/1461765125 learned_addr learned my addr 172.21.15.104:0/1461765125 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): get_auth_request con 0x7f03341365d0 auth_method 0 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0334136200 0x7f0334145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 10 monclient(hunting): get_auth_request con 0x7f0334136200 auth_method 0 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f033259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient(hunting): handle_auth_done global_id 15189 payload 824 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 -- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0334136200 msgr2=0x7f0334145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0334136200 0x7f0334145870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 -- 172.21.15.104:0/1461765125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033413b390 msgr2=0x7f033406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0332d9d700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033413b390 0x7f033406ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033413b390 0x7f033406ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient: _finish_hunting 0 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 monclient: found mon.noname-a 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0331d9b700 1 -- 172.21.15.104:0/1461765125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324009420 con 0x7f03341365d0 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0332d9d700 10 monclient: get_auth_request con 0x7f033413b390 auth_method 0 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.417+0000 7f0332d9d700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033413b390 0x7f033406ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f0331d9b700 1 -- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 msgr2=0x7f0334135220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f0331d9b700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 0x7f0334135220 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f032800ec80 tx=0x7f03280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 1 -- 172.21.15.104:0/1461765125 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032800ca00 con 0x7f03341365d0 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: dump: 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 5 monclient: authenticate success, global_id 15189 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 1 -- 172.21.15.104:0/1461765125 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032800cb60 con 0x7f03341365d0 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 10 monclient: _renew_subs 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0334121330 con 0x7f03341365d0 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 1 -- 172.21.15.104:0/1461765125 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328013470 con 0x7f03341365d0 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f0330d99700 4 set_mon_vals no callback set 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 10 monclient: dump: 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 10 monclient: _renew_subs 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03341203d0 con 0x7f03341365d0 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 1 -- 172.21.15.104:0/1461765125 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f03280137e0 con 0x7f03341365d0 2024-09-18T14:27:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 1 --2- 172.21.15.104:0/1461765125 >> v2:172.21.15.174:6812/17603 conn(0x7f0314035a80 0x7f0314037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f03227fc700 1 -- 172.21.15.104:0/1461765125 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f03280481e0 con 0x7f03341365d0 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033259c700 1 --2- 172.21.15.104:0/1461765125 >> v2:172.21.15.174:6812/17603 conn(0x7f0314035a80 0x7f0314037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033259c700 10 monclient: get_auth_request con 0x7f0314035a80 auth_method 0 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.421+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03340633b0 con 0x7f03341365d0 2024-09-18T14:27:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.425+0000 7f033259c700 1 --2- 172.21.15.104:0/1461765125 >> v2:172.21.15.174:6812/17603 conn(0x7f0314035a80 0x7f0314037f30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f032c006e80 tx=0x7f032c0053c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.429+0000 7f03227fc700 1 -- 172.21.15.104:0/1461765125 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0328023ad0 con 0x7f03341365d0 2024-09-18T14:27:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.429+0000 7f03227fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.429+0000 7f03227fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:53.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.593+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0334068a50 con 0x7f0314035a80 2024-09-18T14:27:53.602 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.597+0000 7f03227fc700 1 -- 172.21.15.104:0/1461765125 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51742 (secure 0 0 0) 0x7f0334068a50 con 0x7f0314035a80 2024-09-18T14:27:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:27:53.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.597+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 >> v2:172.21.15.174:6812/17603 conn(0x7f0314035a80 msgr2=0x7f0314037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.597+0000 7f033359e700 1 --2- 172.21.15.104:0/1461765125 >> v2:172.21.15.174:6812/17603 conn(0x7f0314035a80 0x7f0314037f30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f032c006e80 tx=0x7f032c0053c0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.597+0000 7f033359e700 10 monclient: shutdown 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.597+0000 7f033359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.597+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 msgr2=0x7f0334135220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.597+0000 7f033359e700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 0x7f0334135220 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f032800ec80 tx=0x7f03280091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 shutdown_connections 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 --2- 172.21.15.104:0/1461765125 >> v2:172.21.15.174:6812/17603 conn(0x7f0314035a80 0x7f0314037f30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033413b390 0x7f033406ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03341365d0 0x7f0334135220 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 --2- 172.21.15.104:0/1461765125 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0334136200 0x7f0334145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 >> 172.21.15.104:0/1461765125 conn(0x7f03340a12c0 msgr2=0x7f033409f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 shutdown_connections 2024-09-18T14:27:53.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.601+0000 7f033359e700 1 -- 172.21.15.104:0/1461765125 wait complete. 2024-09-18T14:27:53.607 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:27:53.617 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T14:27:52.516435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4795856,"kb_used_data":557400424,"kb_used_omap":58312,"kb_used_meta":0,"kb_avail":557400424,"statfs":{"total":575688990720,"available":570778034176,"internally_reserved":0,"allocated":570778034176,"data_stored":570778034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59711673,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003101"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403327+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:27:35.403327+0000","last_peered":"2024-09-18T14:27:35.403327+0000","last_clean":"2024-09-18T14:27:35.403327+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:27:35.403327+0000","last_undegraded":"2024-09-18T14:27:35.403327+0000","last_fullsized":"2024-09-18T14:27:35.403327+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.407340+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:27:35.407340+0000","last_peered":"2024-09-18T14:27:35.407340+0000","last_clean":"2024-09-18T14:27:35.407340+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:27:35.407340+0000","last_undegraded":"2024-09-18T14:27:35.407340+0000","last_fullsized":"2024-09-18T14:27:35.407340+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.405428+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:27:35.405428+0000","last_peered":"2024-09-18T14:27:35.405428+0000","last_clean":"2024-09-18T14:27:35.405428+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:27:35.405428+0000","last_undegraded":"2024-09-18T14:27:35.405428+0000","last_fullsized":"2024-09-18T14:27:35.405428+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.401849+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:27:35.401849+0000","last_peered":"2024-09-18T14:27:35.401849+0000","last_clean":"2024-09-18T14:27:35.401849+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:27:35.401849+0000","last_undegraded":"2024-09-18T14:27:35.401849+0000","last_fullsized":"2024-09-18T14:27:35.401849+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":308,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404309+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:27:35.404309+0000","last_peered":"2024-09-18T14:27:35.404309+0000","last_clean":"2024-09-18T14:27:35.404309+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:27:35.404309+0000","last_undegraded":"2024-09-18T14:27:35.404309+0000","last_fullsized":"2024-09-18T14:27:35.404309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":304,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.008862+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:27:35.008862+0000","last_peered":"2024-09-18T14:27:35.008862+0000","last_clean":"2024-09-18T14:27:35.008862+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:27:35.008862+0000","last_undegraded":"2024-09-18T14:27:35.008862+0000","last_fullsized":"2024-09-18T14:27:35.008862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403712+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:27:35.403712+0000","last_peered":"2024-09-18T14:27:35.403712+0000","last_clean":"2024-09-18T14:27:35.403712+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:27:35.403712+0000","last_undegraded":"2024-09-18T14:27:35.403712+0000","last_fullsized":"2024-09-18T14:27:35.403712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":307,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404551+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:27:35.404551+0000","last_peered":"2024-09-18T14:27:35.404551+0000","last_clean":"2024-09-18T14:27:35.404551+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:27:35.404551+0000","last_undegraded":"2024-09-18T14:27:35.404551+0000","last_fullsized":"2024-09-18T14:27:35.404551+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":403,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403858+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:27:35.403858+0000","last_peered":"2024-09-18T14:27:35.403858+0000","last_clean":"2024-09-18T14:27:35.403858+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:27:35.403858+0000","last_undegraded":"2024-09-18T14:27:35.403858+0000","last_fullsized":"2024-09-18T14:27:35.403858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9390,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9615786,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.77400000000000002,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.90200000000000002,"5min":1.204,"15min":1.204},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.55700000000000005,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.746}]},{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47899999999999998},"max":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91400000000000003,"15min":0.89600000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.38500000000000001},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":3.742},"last":1.0549999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.84299999999999997,"15min":0.85899999999999999},"min":{"1min":0.63400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.6240000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.98299999999999998,"15min":0.89500000000000002},"min":{"1min":0.93400000000000005,"5min":0.623,"15min":0.432},"max":{"1min":1.361,"5min":1.927,"15min":1.927},"last":1.1020000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76400000000000001,"15min":0.79500000000000004},"min":{"1min":0.56799999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.2030000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.999,"15min":1.0309999999999999},"min":{"1min":0.74199999999999999,"5min":0.67700000000000005,"15min":0.57499999999999996},"max":{"1min":1.452,"5min":2.0590000000000002,"15min":4.0919999999999996},"last":0.98499999999999999}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76900000000000002,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.944,"5min":1.944,"15min":3.218},"last":0.876},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.92600000000000005,"15min":1.006},"min":{"1min":0.72199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.5779999999999998},"last":0.95299999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799008,"kb_used_data":92900372,"kb_used_omap":9896,"kb_used_meta":0,"kb_avail":92900372,"statfs":{"total":95948165120,"available":95129980928,"internally_reserved":0,"allocated":95129980928,"data_stored":95129980928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10133777,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":1.3260000000000001,"5min":1.988,"15min":1.988},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.91000000000000003,"15min":0.97199999999999998},"min":{"1min":0.72199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.111,"5min":2.012,"15min":2.012},"last":0.78300000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.79300000000000004},"min":{"1min":0.66400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0980000000000001,"5min":2.714,"15min":2.714},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89100000000000001,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93999999999999995,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.65100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.875,"15min":0.91500000000000004},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.083,"5min":1.345,"15min":1.99},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94699999999999995,"15min":1.0109999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.046,"5min":1.361,"15min":2.1240000000000001},"last":0.93799999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.69199999999999995,"15min":0.72799999999999998},"min":{"1min":0.56399999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.222},"last":0.626},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1990000000000001,"15min":1.7769999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73099999999999998,"15min":0.88700000000000001},"min":{"1min":0.56299999999999994,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.012,"5min":1.2070000000000001,"15min":3.0609999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84499999999999997,"15min":0.99099999999999999},"min":{"1min":0.65200000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.075,"5min":1.29,"15min":3.2799999999999998},"last":0.68500000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":9529,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9758155,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81399999999999995,"15min":0.93000000000000005},"min":{"1min":0.56100000000000005,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0109999999999999,"5min":3.8359999999999999,"15min":5.9100000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.98299999999999998},"min":{"1min":0.52500000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0429999999999999,"5min":3.5609999999999999,"15min":6.173},"last":0.59899999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.89600000000000002,"15min":1.034},"min":{"1min":0.53300000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.131,"5min":4.2519999999999998,"15min":5.8789999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.95599999999999996,"15min":0.89400000000000002},"min":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3140000000000001,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":0.755}]},{"osd":2,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90600000000000003,"15min":1.034},"min":{"1min":0.70399999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2250000000000001,"5min":5.9530000000000003,"15min":6.3109999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.95999999999999996,"15min":1.0649999999999999},"min":{"1min":0.71899999999999997,"5min":0.499,"15min":0.499},"max":{"1min":1.0880000000000001,"5min":5.6159999999999997,"15min":6.2350000000000003},"last":0.82699999999999996}]},{"osd":4,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.095,"5min":2.2109999999999999,"15min":3.8420000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79800000000000004,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.99399999999999999,"5min":2.4169999999999998,"15min":5.8410000000000002},"last":0.63900000000000001}]},{"osd":5,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.81599999999999995,"15min":0.93700000000000006},"min":{"1min":0.61199999999999999,"5min":0.53700000000000003,"15min":0.52100000000000002},"max":{"1min":1.167,"5min":1.581,"15min":4.0739999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92500000000000004,"15min":1.01},"min":{"1min":0.628,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.397,"5min":2.6899999999999999,"15min":4.0940000000000003},"last":0.874}]}]},{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799136,"kb_used_data":92900244,"kb_used_omap":9735,"kb_used_meta":0,"kb_avail":92900244,"statfs":{"total":95948165120,"available":95129849856,"internally_reserved":0,"allocated":95129849856,"data_stored":95129849856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9969199,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59399999999999997,"15min":0.56899999999999995},"min":{"1min":0.46700000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.878,"5min":1.849,"15min":1.849},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.68600000000000005,"15min":0.59799999999999998},"min":{"1min":0.52900000000000003,"5min":0.377,"15min":0.376},"max":{"1min":1.375,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78000000000000003,"15min":0.70199999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.206,"5min":4.7640000000000002,"15min":4.7640000000000002},"last":0.627},{"interface":"front","average":{"1min":0.748,"5min":0.84999999999999998,"15min":0.77700000000000002},"min":{"1min":0.56100000000000005,"5min":0.308,"15min":0.308},"max":{"1min":0.92800000000000005,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92900000000000005,"15min":0.89200000000000002},"min":{"1min":0.70499999999999996,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2350000000000001,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.92200000000000004},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.2729999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.93600000000000005}]},{"osd":4,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.91100000000000003,"15min":1.1830000000000001},"min":{"1min":0.69499999999999995,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.2949999999999999,"5min":1.778,"15min":9.0830000000000002},"last":0.996},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95899999999999996,"15min":1.1359999999999999},"min":{"1min":0.75700000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.218,"5min":2.0089999999999999,"15min":7.1399999999999997},"last":0.81599999999999995}]},{"osd":5,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.96099999999999997,"15min":1.1850000000000001},"min":{"1min":0.79400000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.27,"5min":2.0859999999999999,"15min":9.4900000000000002},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.94699999999999995,"15min":1.155},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0369999999999999,"5min":2.2429999999999999,"15min":8.3420000000000005},"last":0.86399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9693,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9926148,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.44700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":1.508,"15min":1.508},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.28599999999999998},"max":{"1min":0.81599999999999995,"5min":1.038,"15min":1.038},"last":0.73499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.251},"max":{"1min":0.92200000000000004,"5min":2.278,"15min":2.278},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74399999999999999,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.70099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.88900000000000001,"15min":1.022},"min":{"1min":0.66000000000000003,"5min":0.41499999999999998,"15min":0.34699999999999998},"max":{"1min":1.0109999999999999,"5min":2.8239999999999998,"15min":5.6699999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88,"5min":0.95099999999999996,"15min":0.84899999999999998},"min":{"1min":0.67900000000000005,"5min":0.53500000000000003,"15min":0.32900000000000001},"max":{"1min":1.196,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.85399999999999998}]},{"osd":4,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88600000000000001,"15min":1.002},"min":{"1min":0.72199999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.079,"5min":1.45,"15min":3.7490000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.92900000000000005,"15min":0.86899999999999999},"min":{"1min":0.65300000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.018,"5min":6.141,"15min":6.141},"last":0.78800000000000003}]},{"osd":5,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.873,"15min":0.98999999999999999},"min":{"1min":0.70899999999999996,"5min":0.46800000000000003,"15min":0.40600000000000003},"max":{"1min":1.111,"5min":1.292,"15min":4.2229999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94199999999999995,"15min":0.91200000000000003},"min":{"1min":0.69899999999999995,"5min":0.46100000000000002,"15min":0.41799999999999998},"max":{"1min":1.198,"5min":1.496,"15min":1.496},"last":0.96499999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640336,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":10067,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10308608,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31},"max":{"1min":0.753,"5min":1.298,"15min":1.298},"last":0.47099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.80600000000000005,"5min":1.48,"15min":1.48},"last":0.505},{"interface":"front","average":{"1min":0.755,"5min":0.71199999999999997,"15min":0.69199999999999995},"min":{"1min":0.51400000000000001,"5min":0.46700000000000003,"15min":0.313},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.872,"15min":0.84499999999999997},"min":{"1min":0.65800000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1759999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.53100000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97499999999999998,"5min":1.212,"15min":1.212},"last":0.82199999999999995}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.97699999999999998},"min":{"1min":0.76900000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2470000000000001,"5min":1.579,"15min":1.9470000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.877,"15min":0.98299999999999998},"min":{"1min":0.627,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":2.476},"last":0.92400000000000004}]},{"osd":5,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.91300000000000003,"15min":0.89000000000000001},"min":{"1min":0.69299999999999995,"5min":0.56399999999999995,"15min":0.56299999999999994},"max":{"1min":1.2989999999999999,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93200000000000005,"15min":0.99399999999999999},"min":{"1min":0.73099999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1539999999999999,"5min":1.3009999999999999,"15min":1.7989999999999999},"last":0.79600000000000004}]}]}],"pool_statfs":[]}} 2024-09-18T14:27:53.619 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient: get_monmap_and_config 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient: build_initial_monmap 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient: monmap: 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:53.720687+0000 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:53.720687+0000 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:53.722 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.723 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:53.723 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:53.723 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:53.723 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:53.723 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:53.723 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 Processor -- start 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 -- start start 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient: init 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f66100a5ca0 0x7f661009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): picked mon.noname-a con 0x7f66100a5ca0 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:53.726 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f661009ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): picked mon.noname-b con 0x7f6610099070 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f66100a04c0 0x7f66100a0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): picked mon.noname-c con 0x7f66100a04c0 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66100a64d0 con 0x7f66100a5ca0 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6610099ff0 con 0x7f66100a04c0 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66100a6070 con 0x7f6610099070 2024-09-18T14:27:53.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f6617c2e700 10 monclient(hunting): authenticate will time out at 1194.710324s 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f661009ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f66100a5ca0 0x7f661009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f661009ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:42974/0 (socket says 172.21.15.104:42974) 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 1 -- 172.21.15.104:0/396150042 learned_addr learned my addr 172.21.15.104:0/396150042 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 10 monclient(hunting): get_auth_request con 0x7f6610099070 auth_method 0 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): get_auth_request con 0x7f66100a5ca0 auth_method 0 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66161cb700 1 --2- 172.21.15.104:0/396150042 >> v2:172.21.15.174:3300/0 conn(0x7f66100a04c0 0x7f66100a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66161cb700 10 monclient(hunting): get_auth_request con 0x7f66100a04c0 auth_method 0 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66161cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66161cb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66161cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient(hunting): handle_auth_done global_id 15192 payload 274 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 1 -- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 msgr2=0x7f661009ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 1 --2- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f661009ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 1 -- 172.21.15.104:0/396150042 >> v2:172.21.15.174:3300/0 conn(0x7f66100a04c0 msgr2=0x7f66100a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 1 --2- 172.21.15.104:0/396150042 >> v2:172.21.15.174:3300/0 conn(0x7f66100a04c0 0x7f66100a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient: _finish_hunting 0 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 1 monclient: found mon.noname-a 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66159ca700 1 -- 172.21.15.104:0/396150042 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661009c4f0 con 0x7f66100a5ca0 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66161cb700 1 --2- 172.21.15.104:0/396150042 >> v2:172.21.15.174:3300/0 conn(0x7f66100a04c0 0x7f66100a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:27:53.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.717+0000 7f66151c9700 1 --2- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f661009ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:53.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66159ca700 1 -- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 msgr2=0x7f661009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66159ca700 1 --2- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661009a9d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f66100a3440 tx=0x7f66000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4fa6476f206ec7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 1 -- 172.21.15.104:0/396150042 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600011840 con 0x7f66100a5ca0 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: dump: 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.737 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: _finish_auth 0 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 5 monclient: authenticate success, global_id 15192 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 1 -- 172.21.15.104:0/396150042 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66000119a0 con 0x7f66100a5ca0 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 4 set_mon_vals no callback set 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 1 -- 172.21.15.104:0/396150042 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600011d40 con 0x7f66100a5ca0 2024-09-18T14:27:53.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f66149c8700 10 monclient: dump: 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.739 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 10 monclient: shutdown 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 -- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 msgr2=0x7f661009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 --2- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661009a9d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f66100a3440 tx=0x7f66000091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 -- 172.21.15.104:0/396150042 shutdown_connections 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 --2- 172.21.15.104:0/396150042 >> v2:172.21.15.174:3300/0 conn(0x7f66100a04c0 0x7f66100a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 --2- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f661009ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 --2- 172.21.15.104:0/396150042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661009a9d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 -- 172.21.15.104:0/396150042 >> 172.21.15.104:0/396150042 conn(0x7f6610069dc0 msgr2=0x7f66100a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 -- 172.21.15.104:0/396150042 shutdown_connections 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 1 -- 172.21.15.104:0/396150042 wait complete. 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 10 monclient: build_initial_monmap 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.721+0000 7f6617c2e700 10 monclient: monmap: 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:53.740 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 Processor -- start 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 -- start start 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient: init 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T14:27:53.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f6610137e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): picked mon.noname-c con 0x7f6610099070 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66100a04c0 0x7f661006b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): picked mon.noname-b con 0x7f66100a04c0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661006bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): picked mon.noname-a con 0x7f66100a5ca0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661006e8f0 con 0x7f66100a5ca0 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661006e610 con 0x7f66100a04c0 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661006cb50 con 0x7f6610099070 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:53.742 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f6617c2e700 10 monclient(hunting): authenticate will time out at 1194.717607s 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661006bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f6610137e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661006bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35692/0 (socket says 172.21.15.104:35692) 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 -- 172.21.15.104:0/2936427534 learned_addr learned my addr 172.21.15.104:0/2936427534 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): get_auth_request con 0x7f66100a5ca0 auth_method 0 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 10 monclient(hunting): get_auth_request con 0x7f6610099070 auth_method 0 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.743 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66151c9700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66100a04c0 0x7f661006b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66159ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66151c9700 10 monclient(hunting): get_auth_request con 0x7f66100a04c0 auth_method 0 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66151c9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66151c9700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66151c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient(hunting): handle_auth_done global_id 15195 payload 824 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 -- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 msgr2=0x7f6610137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.744 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f6610137e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 -- 172.21.15.104:0/2936427534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66100a04c0 msgr2=0x7f661006b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66100a04c0 0x7f661006b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient: _finish_hunting 0 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 monclient: found mon.noname-a 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66161cb700 1 -- 172.21.15.104:0/2936427534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c0094f0 con 0x7f66100a5ca0 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.725+0000 7f66151c9700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66100a04c0 0x7f661006b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f66161cb700 1 -- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 msgr2=0x7f661006bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f66161cb700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661006bdd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f660800d180 tx=0x7f6608009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 1 -- 172.21.15.104:0/2936427534 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608015440 con 0x7f66100a5ca0 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.745 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: dump: 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:53.746 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 1 -- 172.21.15.104:0/2936427534 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660800f040 con 0x7f66100a5ca0 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 5 monclient: authenticate success, global_id 15195 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 10 monclient: _renew_subs 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6600009420 con 0x7f66100a5ca0 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 1 -- 172.21.15.104:0/2936427534 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66080157f0 con 0x7f66100a5ca0 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 10 monclient: dump: 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:53.747 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f66077fe700 4 set_mon_vals no callback set 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 10 monclient: _renew_subs 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6610132380 con 0x7f66100a5ca0 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.748 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661006eab0 con 0x7f66100a5ca0 2024-09-18T14:27:53.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 1 -- 172.21.15.104:0/2936427534 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f660801e3b0 con 0x7f66100a5ca0 2024-09-18T14:27:53.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 1 --2- 172.21.15.104:0/2936427534 >> v2:172.21.15.174:6812/17603 conn(0x7f65f4035a80 0x7f65f4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:53.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f6605ffb700 1 -- 172.21.15.104:0/2936427534 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f66080487e0 con 0x7f66100a5ca0 2024-09-18T14:27:53.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f66159ca700 1 --2- 172.21.15.104:0/2936427534 >> v2:172.21.15.174:6812/17603 conn(0x7f65f4035a80 0x7f65f4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:53.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.729+0000 7f66159ca700 10 monclient: get_auth_request con 0x7f65f4035a80 auth_method 0 2024-09-18T14:27:53.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.733+0000 7f66159ca700 1 --2- 172.21.15.104:0/2936427534 >> v2:172.21.15.174:6812/17603 conn(0x7f65f4035a80 0x7f65f4037f30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f66000091e0 tx=0x7f6600017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:53.749 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.737+0000 7f6605ffb700 1 -- 172.21.15.104:0/2936427534 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6608011a90 con 0x7f66100a5ca0 2024-09-18T14:27:53.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.737+0000 7f6605ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:53.750 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.737+0000 7f6605ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:53.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.905+0000 7f6617c2e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:27:53.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.905+0000 7f6617c2e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:27:53.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.905+0000 7f6617c2e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:53.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.905+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66100636d0 con 0x7f66100a5ca0 2024-09-18T14:27:53.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.905+0000 7f6605ffb700 1 -- 172.21.15.104:0/2936427534 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v327) v1 ==== 74+0+10033 (secure 0 0 0) 0x7f660801a020 con 0x7f66100a5ca0 2024-09-18T14:27:53.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.905+0000 7f6605ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T14:27:53.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.905+0000 7f6605ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:27:53.912 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:27:53.912 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":327,"fsid":"ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8","created":"2024-09-18T14:21:19.516787+0000","modified":"2024-09-18T14:27:37.789105+0000","last_up_change":"2024-09-18T14:21:25.123789+0000","last_in_change":"2024-09-18T14:21:22.111706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T14:21:26.474399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T14:21:30.433492+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f9ec9d12-b31d-49ef-a56d-c7fa21e03aad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":17763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6809","nonce":17763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6811","nonce":17763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":17763}]},"public_addr":"172.21.15.104:6808/17763","cluster_addr":"172.21.15.104:6809/17763","heartbeat_back_addr":"172.21.15.104:6811/17763","heartbeat_front_addr":"172.21.15.104:6810/17763","state":["exists","up"]},{"osd":1,"uuid":"9522965d-f7f2-4d3e-8c54-e2377d2bfb56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":17761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6805","nonce":17761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6807","nonce":17761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":17761}]},"public_addr":"172.21.15.104:6804/17761","cluster_addr":"172.21.15.104:6805/17761","heartbeat_back_addr":"172.21.15.104:6807/17761","heartbeat_front_addr":"172.21.15.104:6806/17761","state":["exists","up"]},{"osd":2,"uuid":"07bd6e83-04fd-43e8-b538-9ee89fdbf1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":17762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6801","nonce":17762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6803","nonce":17762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":17762}]},"public_addr":"172.21.15.104:6800/17762","cluster_addr":"172.21.15.104:6801/17762","heartbeat_back_addr":"172.21.15.104:6803/17762","heartbeat_front_addr":"172.21.15.104:6802/17762","state":["exists","up"]},{"osd":3,"uuid":"2876367d-d252-4764-b433-9cd2c82448b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":17845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":17845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":17845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":17845}]},"public_addr":"172.21.15.174:6808/17845","cluster_addr":"172.21.15.174:6809/17845","heartbeat_back_addr":"172.21.15.174:6811/17845","heartbeat_front_addr":"172.21.15.174:6810/17845","state":["exists","up"]},{"osd":4,"uuid":"53d9c8fe-0cd6-49aa-94ed-13e319f7b8f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":17847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6801","nonce":17847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":17847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":17847}]},"public_addr":"172.21.15.174:6800/17847","cluster_addr":"172.21.15.174:6801/17847","heartbeat_back_addr":"172.21.15.174:6803/17847","heartbeat_front_addr":"172.21.15.174:6802/17847","state":["exists","up"]},{"osd":5,"uuid":"57b7828f-2f7a-431b-8ed4-726d1011dd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":324,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":17846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":17846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":17846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":17846}]},"public_addr":"172.21.15.174:6804/17846","cluster_addr":"172.21.15.174:6805/17846","heartbeat_back_addr":"172.21.15.174:6807/17846","heartbeat_front_addr":"172.21.15.174:6806/17846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.237413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.242543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.214389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.213881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.183415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:21:23.230911+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:27:53.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 >> v2:172.21.15.174:6812/17603 conn(0x7f65f4035a80 msgr2=0x7f65f4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 --2- 172.21.15.104:0/2936427534 >> v2:172.21.15.174:6812/17603 conn(0x7f65f4035a80 0x7f65f4037f30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f66000091e0 tx=0x7f6600017000 comp rx=0 tx=0).stop 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 10 monclient: shutdown 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 msgr2=0x7f661006bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661006bdd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f660800d180 tx=0x7f6608009a50 comp rx=0 tx=0).stop 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 shutdown_connections 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 --2- 172.21.15.104:0/2936427534 >> v2:172.21.15.174:6812/17603 conn(0x7f65f4035a80 0x7f65f4037f30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66100a5ca0 0x7f661006bdd0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66100a04c0 0x7f661006b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 --2- 172.21.15.104:0/2936427534 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f6610099070 0x7f6610137e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 >> 172.21.15.104:0/2936427534 conn(0x7f6610069dc0 msgr2=0x7f661009a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 shutdown_connections 2024-09-18T14:27:53.915 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:53.909+0000 7f6617c2e700 1 -- 172.21.15.104:0/2936427534 wait complete. 2024-09-18T14:27:54.927 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T14:27:54.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient: get_monmap_and_config 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient: monmap: 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:55.048430+0000 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:55.048430+0000 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.050 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.051 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:55.051 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:55.051 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:55.051 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:55.051 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:55.051 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 1 Processor -- start 2024-09-18T14:27:55.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 1 -- start start 2024-09-18T14:27:55.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient: init 2024-09-18T14:27:55.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T14:27:55.054 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa0bc09bf40 0x7fa0bc09ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient(hunting): picked mon.noname-c con 0x7fa0bc09bf40 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0bc09ff00 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa0bc0a09e0 0x7fa0bc1396a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient(hunting): picked mon.noname-a con 0x7fa0bc0a09e0 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0bc09a760 con 0x7fa0bc09ff00 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc09a480 con 0x7fa0bc0a09e0 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc09c310 con 0x7fa0bc09bf40 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c2246700 10 monclient(hunting): authenticate will time out at 1196.038590s 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa0bc0a09e0 0x7fa0bc1396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.055 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa0bc0a09e0 0x7fa0bc1396a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35706/0 (socket says 172.21.15.104:35706) 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 -- 172.21.15.104:0/705379224 learned_addr learned my addr 172.21.15.104:0/705379224 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): get_auth_request con 0x7fa0bc0a09e0 auth_method 0 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 1 --2- 172.21.15.104:0/705379224 >> v2:172.21.15.174:3300/0 conn(0x7fa0bc09bf40 0x7fa0bc09ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 10 monclient(hunting): get_auth_request con 0x7fa0bc09bf40 auth_method 0 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.056 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient(hunting): handle_auth_done global_id 15198 payload 274 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 -- 172.21.15.104:0/705379224 >> v2:172.21.15.174:3300/0 conn(0x7fa0bc09bf40 msgr2=0x7fa0bc09ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 --2- 172.21.15.104:0/705379224 >> v2:172.21.15.174:3300/0 conn(0x7fa0bc09bf40 0x7fa0bc09ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 -- 172.21.15.104:0/705379224 >> v1:172.21.15.174:6789/0 conn(0x7fa0bc09ff00 legacy=0x7fa0bc0a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 monclient: found mon.noname-a 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.057 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.045+0000 7fa0c1a45700 1 -- 172.21.15.104:0/705379224 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0a1e00 con 0x7fa0bc0a09e0 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c1a45700 1 -- 172.21.15.104:0/705379224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0a09e0 msgr2=0x7fa0bc1396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c1a45700 1 --2- 172.21.15.104:0/705379224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0a09e0 0x7fa0bc1396a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc06b4b0 tx=0x7fa0b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7637c524b792fb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 1 -- 172.21.15.104:0/705379224 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b400e070 con 0x7fa0bc0a09e0 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: dump: 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 5 monclient: authenticate success, global_id 15198 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 1 -- 172.21.15.104:0/705379224 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b400bc30 con 0x7fa0bc0a09e0 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 4 set_mon_vals no callback set 2024-09-18T14:27:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 1 -- 172.21.15.104:0/705379224 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b40092e0 con 0x7fa0bc0a09e0 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0b3fff700 10 monclient: dump: 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 10 monclient: shutdown 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 -- 172.21.15.104:0/705379224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0a09e0 msgr2=0x7fa0bc1396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 --2- 172.21.15.104:0/705379224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0a09e0 0x7fa0bc1396a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc06b4b0 tx=0x7fa0b4009840 comp rx=0 tx=0).stop 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 -- 172.21.15.104:0/705379224 shutdown_connections 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 --2- 172.21.15.104:0/705379224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0a09e0 0x7fa0bc1396a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 --2- 172.21.15.104:0/705379224 >> v2:172.21.15.174:3300/0 conn(0x7fa0bc09bf40 0x7fa0bc09ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 -- 172.21.15.104:0/705379224 >> 172.21.15.104:0/705379224 conn(0x7fa0bc0a5510 msgr2=0x7fa0bc0a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 -- 172.21.15.104:0/705379224 shutdown_connections 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 1 -- 172.21.15.104:0/705379224 wait complete. 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.049+0000 7fa0c2246700 10 monclient: monmap: 2024-09-18T14:27:55.065 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 Processor -- start 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 -- start start 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient: init 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa0bc09bf40 0x7fa0bc08db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): picked mon.noname-c con 0x7fa0bc09bf40 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc09ff00 0x7fa0bc08e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): picked mon.noname-a con 0x7fa0bc09ff00 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0bc0a09e0 0x7fa0bc08e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): picked mon.noname-b con 0x7fa0bc0a09e0 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc0995d0 con 0x7fa0bc09ff00 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc09eee0 con 0x7fa0bc0a09e0 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc098650 con 0x7fa0bc09bf40 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c2246700 10 monclient(hunting): authenticate will time out at 1196.044929s 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc09ff00 0x7fa0bc08e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa0bc09bf40 0x7fa0bc08db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc09ff00 0x7fa0bc08e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35712/0 (socket says 172.21.15.104:35712) 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 1 -- 172.21.15.104:0/2189906373 learned_addr learned my addr 172.21.15.104:0/2189906373 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 10 monclient(hunting): get_auth_request con 0x7fa0bc09ff00 auth_method 0 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): get_auth_request con 0x7fa0bc09bf40 auth_method 0 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1a45700 1 --2- 172.21.15.104:0/2189906373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0bc0a09e0 0x7fa0bc08e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1a45700 10 monclient(hunting): get_auth_request con 0x7fa0bc0a09e0 auth_method 0 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1a45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1a45700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1a45700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient(hunting): handle_auth_done global_id 34151 payload 824 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 1 -- 172.21.15.104:0/2189906373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0bc0a09e0 msgr2=0x7fa0bc08e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 1 --2- 172.21.15.104:0/2189906373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0bc0a09e0 0x7fa0bc08e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 1 -- 172.21.15.104:0/2189906373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc09ff00 msgr2=0x7fa0bc08e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 1 --2- 172.21.15.104:0/2189906373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc09ff00 0x7fa0bc08e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 1 monclient: found mon.noname-c 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1244700 1 -- 172.21.15.104:0/2189906373 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b4009ae0 con 0x7fa0bc09bf40 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c1a45700 1 --2- 172.21.15.104:0/2189906373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0bc0a09e0 0x7fa0bc08e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.053+0000 7fa0c0a43700 1 --2- 172.21.15.104:0/2189906373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc09ff00 0x7fa0bc08e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c1244700 1 -- 172.21.15.104:0/2189906373 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa0bc09bf40 msgr2=0x7fa0bc08db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c1244700 1 --2- 172.21.15.104:0/2189906373 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa0bc09bf40 0x7fa0bc08db00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00ea80 tx=0x7fa0ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0095d0 con 0x7fa0bc09bf40 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: dump: 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 5 monclient: authenticate success, global_id 34151 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac00ca00 con 0x7fa0bc09bf40 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 10 monclient: _renew_subs 2024-09-18T14:27:55.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc099040 con 0x7fa0bc09bf40 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac00cdc0 con 0x7fa0bc09bf40 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b37fe700 4 set_mon_vals no callback set 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: dump: 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.072 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.073 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.073 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 10 monclient: _renew_subs 2024-09-18T14:27:55.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:55.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc090140 con 0x7fa0bc09bf40 2024-09-18T14:27:55.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:55.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0bc064b30 con 0x7fa0bc09bf40 2024-09-18T14:27:55.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fa0ac013680 con 0x7fa0bc09bf40 2024-09-18T14:27:55.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.174:6812/17603 conn(0x7fa0a0035a80 0x7fa0a0037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa0ac0484d0 con 0x7fa0bc09bf40 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.104:6808/17763 conn(0x7fa0a003b5a0 0x7fa0a003d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 --> v2:172.21.15.104:6808/17763 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0a003e020 con 0x7fa0a003b5a0 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_get_version_reply(handle=1 version=327) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0ac04aa60 con 0x7fa0bc09bf40 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0b17fa700 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c0a43700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.174:6812/17603 conn(0x7fa0a0035a80 0x7fa0a0037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c1a45700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.104:6808/17763 conn(0x7fa0a003b5a0 0x7fa0a003d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c0a43700 10 monclient: get_auth_request con 0x7fa0a0035a80 auth_method 0 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.057+0000 7fa0c1a45700 10 monclient: get_auth_request con 0x7fa0a003b5a0 auth_method 0 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.061+0000 7fa0c0a43700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.174:6812/17603 conn(0x7fa0a0035a80 0x7fa0a0037f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8007fd0 tx=0x7fa0b8007f60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.065+0000 7fa0c1a45700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.104:6808/17763 conn(0x7fa0a003b5a0 0x7fa0a003d9b0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.065+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== osd.0 v2:172.21.15.104:6808/17763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fa0a003e020 con 0x7fa0a003b5a0 2024-09-18T14:27:55.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.081+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 --> v2:172.21.15.104:6808/17763 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa0bc08ed00 con 0x7fa0a003b5a0 2024-09-18T14:27:55.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.081+0000 7fa0b17fa700 1 -- 172.21.15.104:0/2189906373 <== osd.0 v2:172.21.15.104:6808/17763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa0bc08ed00 con 0x7fa0a003b5a0 2024-09-18T14:27:55.089 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:55.089 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:27:55.089 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 >> v2:172.21.15.104:6808/17763 conn(0x7fa0a003b5a0 msgr2=0x7fa0a003d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.104:6808/17763 conn(0x7fa0a003b5a0 0x7fa0a003d9b0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 >> v2:172.21.15.174:6812/17603 conn(0x7fa0a0035a80 msgr2=0x7fa0a0037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 --2- 172.21.15.104:0/2189906373 >> v2:172.21.15.174:6812/17603 conn(0x7fa0a0035a80 0x7fa0a0037f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8007fd0 tx=0x7fa0b8007f60 comp rx=0 tx=0).stop 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 10 monclient: shutdown 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa0bc09bf40 msgr2=0x7fa0bc08db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 --2- 172.21.15.104:0/2189906373 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fa0bc09bf40 0x7fa0bc08db00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00ea80 tx=0x7fa0ac0091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c1a45700 1 -- 172.21.15.104:0/2189906373 reap_dead start 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 shutdown_connections 2024-09-18T14:27:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 >> 172.21.15.104:0/2189906373 conn(0x7fa0bc0a5510 msgr2=0x7fa0bc09e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 shutdown_connections 2024-09-18T14:27:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.085+0000 7fa0c2246700 1 -- 172.21.15.104:0/2189906373 wait complete. 2024-09-18T14:27:55.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T14:27:55.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.209+0000 7f0317243700 10 monclient: get_monmap_and_config 2024-09-18T14:27:55.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.209+0000 7f0317243700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.209+0000 7f0317243700 10 monclient: monmap: 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:55.213008+0000 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:55.213008+0000 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:55.215 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:55.216 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.209+0000 7f0317243700 1 Processor -- start 2024-09-18T14:27:55.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.209+0000 7f0317243700 1 -- start start 2024-09-18T14:27:55.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.209+0000 7f0317243700 10 monclient: init 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.209+0000 7f0317243700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03100a23e0 0x7f031006b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient(hunting): picked mon.noname-a con 0x7f03100a23e0 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f031006ee90 0x7f031006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient(hunting): picked mon.noname-c con 0x7f031006ee90 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f031006c300 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031006d260 con 0x7f031006c300 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031006c6d0 con 0x7f03100a23e0 2024-09-18T14:27:55.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03100a27b0 con 0x7f031006ee90 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0317243700 10 monclient(hunting): authenticate will time out at 1196.204232s 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03100a23e0 0x7f031006b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03100a23e0 0x7f031006b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35724/0 (socket says 172.21.15.104:35724) 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 -- 172.21.15.104:0/3919189977 learned_addr learned my addr 172.21.15.104:0/3919189977 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): get_auth_request con 0x7f03100a23e0 auth_method 0 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 1 --2- 172.21.15.104:0/3919189977 >> v2:172.21.15.174:3300/0 conn(0x7f031006ee90 0x7f031006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 10 monclient(hunting): get_auth_request con 0x7f031006ee90 auth_method 0 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient(hunting): handle_auth_done global_id 15204 payload 274 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 -- 172.21.15.104:0/3919189977 >> v2:172.21.15.174:3300/0 conn(0x7f031006ee90 msgr2=0x7f031006bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 --2- 172.21.15.104:0/3919189977 >> v2:172.21.15.174:3300/0 conn(0x7f031006ee90 0x7f031006bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.221 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 -- 172.21.15.104:0/3919189977 >> v1:172.21.15.174:6789/0 conn(0x7f031006c300 legacy=0x7f03101352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:55.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 monclient: found mon.noname-a 2024-09-18T14:27:55.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 -- 172.21.15.104:0/3919189977 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031006d820 con 0x7f03100a23e0 2024-09-18T14:27:55.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 -- 172.21.15.104:0/3919189977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03100a23e0 msgr2=0x7f031006b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f0314fdf700 1 --2- 172.21.15.104:0/3919189977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03100a23e0 0x7f031006b880 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f03040099b0 tx=0x7f0304009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34de5b68110d5d97 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.213+0000 7f030f7fe700 1 -- 172.21.15.104:0/3919189977 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304009ea0 con 0x7f03100a23e0 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: dump: 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.227 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 5 monclient: authenticate success, global_id 15204 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 1 -- 172.21.15.104:0/3919189977 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030400ba30 con 0x7f03100a23e0 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 1 -- 172.21.15.104:0/3919189977 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030400bdf0 con 0x7f03100a23e0 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 4 set_mon_vals no callback set 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030f7fe700 10 monclient: dump: 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.228 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient: shutdown 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- 172.21.15.104:0/3919189977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03100a23e0 msgr2=0x7f031006b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 --2- 172.21.15.104:0/3919189977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03100a23e0 0x7f031006b880 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f03040099b0 tx=0x7f0304009250 comp rx=0 tx=0).stop 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- 172.21.15.104:0/3919189977 shutdown_connections 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 --2- 172.21.15.104:0/3919189977 >> v2:172.21.15.174:3300/0 conn(0x7f031006ee90 0x7f031006bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 --2- 172.21.15.104:0/3919189977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03100a23e0 0x7f031006b880 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- 172.21.15.104:0/3919189977 >> 172.21.15.104:0/3919189977 conn(0x7f0310067e50 msgr2=0x7f03100a1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- 172.21.15.104:0/3919189977 shutdown_connections 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- 172.21.15.104:0/3919189977 wait complete. 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient: monmap: 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.230 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 Processor -- start 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- start start 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient: init 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006c300 0x7f031013dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): picked mon.noname-b con 0x7f031006c300 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 0x7f031013d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): picked mon.noname-a con 0x7f031006ee90 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f03100a23e0 0x7f0310141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): picked mon.noname-c con 0x7f03100a23e0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310122eb0 con 0x7f031006ee90 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101228f0 con 0x7f031006c300 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0310122610 con 0x7f03100a23e0 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0317243700 10 monclient(hunting): authenticate will time out at 1196.211621s 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 0x7f031013d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f03157e0700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f03100a23e0 0x7f0310141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 0x7f031013d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35730/0 (socket says 172.21.15.104:35730) 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030ffff700 1 -- 172.21.15.104:0/2105715162 learned_addr learned my addr 172.21.15.104:0/2105715162 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030ffff700 10 monclient(hunting): get_auth_request con 0x7f031006ee90 auth_method 0 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f030ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f03157e0700 10 monclient(hunting): get_auth_request con 0x7f03100a23e0 auth_method 0 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f03157e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f03157e0700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f03157e0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.217+0000 7f0314fdf700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006c300 0x7f031013dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0314fdf700 10 monclient(hunting): get_auth_request con 0x7f031006c300 auth_method 0 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0314fdf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0314fdf700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0314fdf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f03157e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f03157e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f03157e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0314fdf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0314fdf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0314fdf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 10 monclient(hunting): handle_auth_done global_id 15207 payload 824 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 -- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f03100a23e0 msgr2=0x7f0310141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f03100a23e0 0x7f0310141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 -- 172.21.15.104:0/2105715162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006c300 msgr2=0x7f031013dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006c300 0x7f031013dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 monclient: found mon.noname-a 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 -- 172.21.15.104:0/2105715162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040094f0 con 0x7f031006ee90 2024-09-18T14:27:55.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f03157e0700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f03100a23e0 0x7f0310141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 -- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 msgr2=0x7f031013d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030ffff700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 0x7f031013d570 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00ee10 tx=0x7f02fc00b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 1 -- 172.21.15.104:0/2105715162 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc010440 con 0x7f031006ee90 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: dump: 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.237 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 1 -- 172.21.15.104:0/2105715162 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc0105a0 con 0x7f031006ee90 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0317243700 5 monclient: authenticate success, global_id 15207 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0317243700 10 monclient: _renew_subs 2024-09-18T14:27:55.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0317243700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0310122bd0 con 0x7f031006ee90 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 1 -- 172.21.15.104:0/2105715162 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc010960 con 0x7f031006ee90 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030cff9700 10 monclient: dump: 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.239 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f030effd700 4 set_mon_vals no callback set 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0317243700 10 monclient: _renew_subs 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0317243700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.221+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031013dab0 con 0x7f031006ee90 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f0317243700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f0317243700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f0317243700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03100636d0 con 0x7f031006ee90 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f030cff9700 1 -- 172.21.15.104:0/2105715162 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f02fc010d50 con 0x7f031006ee90 2024-09-18T14:27:55.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f030cff9700 1 --2- 172.21.15.104:0/2105715162 >> v2:172.21.15.174:6812/17603 conn(0x7f02f4035ad0 0x7f02f4037f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f030cff9700 1 -- 172.21.15.104:0/2105715162 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f02fc014030 con 0x7f031006ee90 2024-09-18T14:27:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f0314fdf700 1 --2- 172.21.15.104:0/2105715162 >> v2:172.21.15.174:6812/17603 conn(0x7f02f4035ad0 0x7f02f4037f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.225+0000 7f0314fdf700 10 monclient: get_auth_request con 0x7f02f4035ad0 auth_method 0 2024-09-18T14:27:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.229+0000 7f0314fdf700 1 --2- 172.21.15.104:0/2105715162 >> v2:172.21.15.174:6812/17603 conn(0x7f02f4035ad0 0x7f02f4037f80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f03040099b0 tx=0x7f0304016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.233+0000 7f030cff9700 1 -- 172.21.15.104:0/2105715162 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f02fc01f8a0 con 0x7f031006ee90 2024-09-18T14:27:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.233+0000 7f030cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:55.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.233+0000 7f030cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.397+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f0310123020 con 0x7f02f4035ad0 2024-09-18T14:27:55.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f030cff9700 1 -- 172.21.15.104:0/2105715162 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0310123020 con 0x7f02f4035ad0 2024-09-18T14:27:55.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 >> v2:172.21.15.174:6812/17603 conn(0x7f02f4035ad0 msgr2=0x7f02f4037f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 --2- 172.21.15.104:0/2105715162 >> v2:172.21.15.174:6812/17603 conn(0x7f02f4035ad0 0x7f02f4037f80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f03040099b0 tx=0x7f0304016000 comp rx=0 tx=0).stop 2024-09-18T14:27:55.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 10 monclient: shutdown 2024-09-18T14:27:55.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 msgr2=0x7f031013d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 0x7f031013d570 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00ee10 tx=0x7f02fc00b430 comp rx=0 tx=0).stop 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 shutdown_connections 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 --2- 172.21.15.104:0/2105715162 >> v2:172.21.15.174:6812/17603 conn(0x7f02f4035ad0 0x7f02f4037f80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f03100a23e0 0x7f0310141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f031006ee90 0x7f031013d570 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 --2- 172.21.15.104:0/2105715162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f031006c300 0x7f031013dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 >> 172.21.15.104:0/2105715162 conn(0x7f0310067e50 msgr2=0x7f031009a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 shutdown_connections 2024-09-18T14:27:55.409 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.401+0000 7f0317243700 1 -- 172.21.15.104:0/2105715162 wait complete. 2024-09-18T14:27:55.410 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T14:27:55.419 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T14:27:55.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:27:55.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient: get_monmap_and_config 2024-09-18T14:27:55.530 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient: monmap: 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:55.529146+0000 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:55.529146+0000 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:55.531 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:55.532 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 1 Processor -- start 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 1 -- start start 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient: init 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c140a29a0 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c1409a700 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9c14138930 0x7f9c1413ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient(hunting): picked mon.noname-c con 0x7f9c14138930 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c14068c10 con 0x7f9c140a29a0 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140a2d70 con 0x7f9c1409a700 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1406b640 con 0x7f9c14138930 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1c314700 10 monclient(hunting): authenticate will time out at 1196.519174s 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a0b0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9c140a29a0 0x7f9c140a11a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34822/0 (socket says 172.21.15.104:34822) 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a0b0700 1 -- 172.21.15.104:0/427442022 learned_addr learned my addr 172.21.15.104:0/427442022 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 1 --2- 172.21.15.104:0/427442022 >> v2:172.21.15.174:3300/0 conn(0x7f9c14138930 0x7f9c1413ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 10 monclient(hunting): get_auth_request con 0x7f9c14138930 auth_method 0 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.525+0000 7f9c1a8b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 10 monclient(hunting): handle_auth_done global_id 5854 payload 274 2024-09-18T14:27:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 1 -- 172.21.15.104:0/427442022 >> v1:172.21.15.174:6789/0 conn(0x7f9c1409a700 legacy=0x7f9c140a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 1 -- 172.21.15.104:0/427442022 >> v1:172.21.15.104:6789/0 conn(0x7f9c140a29a0 legacy=0x7f9c140a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 1 monclient: found mon.noname-c 2024-09-18T14:27:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:27:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 1 -- 172.21.15.104:0/427442022 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140a1fc0 con 0x7f9c14138930 2024-09-18T14:27:55.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 1 -- 172.21.15.104:0/427442022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 msgr2=0x7f9c1413ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1a8b1700 1 --2- 172.21.15.104:0/427442022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 0x7f9c1413ad10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009e60 tx=0x7f9c04009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=195ecc1657d6ddae server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 1 -- 172.21.15.104:0/427442022 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400e070 con 0x7f9c14138930 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: dump: 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 5 monclient: authenticate success, global_id 5854 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 1 -- 172.21.15.104:0/427442022 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c04012430 con 0x7f9c14138930 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 1 -- 172.21.15.104:0/427442022 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c040127d0 con 0x7f9c14138930 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 4 set_mon_vals no callback set 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.539 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c190ae700 10 monclient: dump: 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 10 monclient: shutdown 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 1 -- 172.21.15.104:0/427442022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 msgr2=0x7f9c1413ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.529+0000 7f9c1c314700 1 --2- 172.21.15.104:0/427442022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 0x7f9c1413ad10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009e60 tx=0x7f9c04009780 comp rx=0 tx=0).stop 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- 172.21.15.104:0/427442022 shutdown_connections 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 --2- 172.21.15.104:0/427442022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 0x7f9c1413ad10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- 172.21.15.104:0/427442022 >> 172.21.15.104:0/427442022 conn(0x7f9c140a61f0 msgr2=0x7f9c140a2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- 172.21.15.104:0/427442022 shutdown_connections 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- 172.21.15.104:0/427442022 wait complete. 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient: monmap: 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.541 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.542 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:55.542 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:55.542 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:55.542 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 Processor -- start 2024-09-18T14:27:55.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- start start 2024-09-18T14:27:55.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient: init 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c1409a700 0x7f9c14141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): picked mon.noname-a con 0x7f9c1409a700 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c140a29a0 0x7f9c14134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): picked mon.noname-c con 0x7f9c140a29a0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 0x7f9c1406df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): picked mon.noname-b con 0x7f9c14138930 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1412ae10 con 0x7f9c1409a700 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140a3d40 con 0x7f9c14138930 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1412afe0 con 0x7f9c140a29a0 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1c314700 10 monclient(hunting): authenticate will time out at 1196.526147s 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c1409a700 0x7f9c14141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c140a29a0 0x7f9c14134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c1409a700 0x7f9c14141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35734/0 (socket says 172.21.15.104:35734) 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 -- 172.21.15.104:0/4289386328 learned_addr learned my addr 172.21.15.104:0/4289386328 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 10 monclient(hunting): get_auth_request con 0x7f9c140a29a0 auth_method 0 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): get_auth_request con 0x7f9c1409a700 auth_method 0 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 1 --2- 172.21.15.104:0/4289386328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 0x7f9c1406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 10 monclient(hunting): get_auth_request con 0x7f9c14138930 auth_method 0 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a8b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient(hunting): handle_auth_done global_id 15210 payload 824 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 -- 172.21.15.104:0/4289386328 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c140a29a0 msgr2=0x7f9c14134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 --2- 172.21.15.104:0/4289386328 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c140a29a0 0x7f9c14134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 -- 172.21.15.104:0/4289386328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 msgr2=0x7f9c1406df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 --2- 172.21.15.104:0/4289386328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c14138930 0x7f9c1406df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 monclient: found mon.noname-a 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c1a0b0700 1 -- 172.21.15.104:0/4289386328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04009a20 con 0x7f9c1409a700 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.533+0000 7f9c198af700 1 --2- 172.21.15.104:0/4289386328 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f9c140a29a0 0x7f9c14134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1a0b0700 1 -- 172.21.15.104:0/4289386328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c1409a700 msgr2=0x7f9c14141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1a0b0700 1 --2- 172.21.15.104:0/4289386328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c1409a700 0x7f9c14141000 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00c5e0 tx=0x7f9c0c004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c005d40 con 0x7f9c1409a700 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: dump: 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.553 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 5 monclient: authenticate success, global_id 15210 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c0070d0 con 0x7f9c1409a700 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 10 monclient: _renew_subs 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1412bed0 con 0x7f9c1409a700 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c007490 con 0x7f9c1409a700 2024-09-18T14:27:55.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 10 monclient: dump: 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c188ad700 4 set_mon_vals no callback set 2024-09-18T14:27:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 10 monclient: _renew_subs 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14135250 con 0x7f9c1409a700 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9c14064d50 con 0x7f9c1409a700 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f9c0c01b3b0 con 0x7f9c1409a700 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.174:6812/17603 conn(0x7f9c00035a80 0x7f9c00037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.537+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9c0c047340 con 0x7f9c1409a700 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c127fc700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.104:6804/17761 conn(0x7f9c0003b5a0 0x7f9c0003d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 --> v2:172.21.15.104:6804/17761 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c0003e020 con 0x7f9c0003b5a0 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=327) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c0c049870 con 0x7f9c1409a700 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c127fc700 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c1a8b1700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.104:6804/17761 conn(0x7f9c0003b5a0 0x7f9c0003d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c1a8b1700 10 monclient: get_auth_request con 0x7f9c0003b5a0 auth_method 0 2024-09-18T14:27:55.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c198af700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.174:6812/17603 conn(0x7f9c00035a80 0x7f9c00037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c198af700 10 monclient: get_auth_request con 0x7f9c00035a80 auth_method 0 2024-09-18T14:27:55.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.541+0000 7f9c198af700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.174:6812/17603 conn(0x7f9c00035a80 0x7f9c00037f30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9c08002e00 tx=0x7f9c08007ea0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.545+0000 7f9c1a8b1700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.104:6804/17761 conn(0x7f9c0003b5a0 0x7f9c0003d9b0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.545+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== osd.1 v2:172.21.15.104:6804/17761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f9c0003e020 con 0x7f9c0003b5a0 2024-09-18T14:27:55.568 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.561+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 --> v2:172.21.15.104:6804/17761 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9c14068e70 con 0x7f9c0003b5a0 2024-09-18T14:27:55.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c127fc700 1 -- 172.21.15.104:0/4289386328 <== osd.1 v2:172.21.15.104:6804/17761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9c14068e70 con 0x7f9c0003b5a0 2024-09-18T14:27:55.569 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:55.569 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:27:55.569 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:55.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 >> v2:172.21.15.104:6804/17761 conn(0x7f9c0003b5a0 msgr2=0x7f9c0003d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.104:6804/17761 conn(0x7f9c0003b5a0 0x7f9c0003d9b0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 >> v2:172.21.15.174:6812/17603 conn(0x7f9c00035a80 msgr2=0x7f9c00037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 --2- 172.21.15.104:0/4289386328 >> v2:172.21.15.174:6812/17603 conn(0x7f9c00035a80 0x7f9c00037f30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9c08002e00 tx=0x7f9c08007ea0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 10 monclient: shutdown 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c1409a700 msgr2=0x7f9c14141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 --2- 172.21.15.104:0/4289386328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c1409a700 0x7f9c14141000 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00c5e0 tx=0x7f9c0c004a80 comp rx=0 tx=0).stop 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1a8b1700 1 -- 172.21.15.104:0/4289386328 reap_dead start 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 shutdown_connections 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 >> 172.21.15.104:0/4289386328 conn(0x7f9c140a61f0 msgr2=0x7f9c14137be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 shutdown_connections 2024-09-18T14:27:55.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.565+0000 7f9c1c314700 1 -- 172.21.15.104:0/4289386328 wait complete. 2024-09-18T14:27:55.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T14:27:55.692 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.685+0000 7f2f0b1ca700 10 monclient: get_monmap_and_config 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.685+0000 7f2f0b1ca700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.685+0000 7f2f0b1ca700 10 monclient: monmap: 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:55.691160+0000 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:55.691160+0000 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:55.693 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:55.694 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 1 Processor -- start 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 1 -- start start 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient: init 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f0406f1c0 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f0406f590 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2f0406f960 0x7f2f0413eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient(hunting): picked mon.noname-a con 0x7f2f0406f960 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f04099d70 con 0x7f2f0406f590 2024-09-18T14:27:55.695 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f04099a90 con 0x7f2f0406f1c0 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0409b5e0 con 0x7f2f0406f960 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0b1ca700 10 monclient(hunting): authenticate will time out at 1196.680686s 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f099c7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2f0406f590 0x7f2f0413b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34826/0 (socket says 172.21.15.104:34826) 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f099c7700 1 -- 172.21.15.104:0/4012735060 learned_addr learned my addr 172.21.15.104:0/4012735060 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 1 --2- 172.21.15.104:0/4012735060 >> v2:172.21.15.104:3300/0 conn(0x7f2f0406f960 0x7f2f0413eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): get_auth_request con 0x7f2f0406f960 auth_method 0 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.696 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f099c7700 1 -- 172.21.15.104:0/4012735060 >> v1:172.21.15.104:6789/0 conn(0x7f2f0406f590 legacy=0x7f2f0413b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.697 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient(hunting): handle_auth_done global_id 15213 payload 274 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a1c8700 1 -- 172.21.15.104:0/4012735060 >> v1:172.21.15.174:6789/0 conn(0x7f2f0406f1c0 legacy=0x7f2f0409aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 1 -- 172.21.15.104:0/4012735060 >> v1:172.21.15.174:6789/0 conn(0x7f2f0406f1c0 legacy=0x7f2f0409aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 1 -- 172.21.15.104:0/4012735060 >> v1:172.21.15.104:6789/0 conn(0x7f2f0406f590 legacy=0x7f2f0413b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f091c6700 1 -- 172.21.15.104:0/4012735060 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073569980 0 0) 0x7f2f04099d70 con 0x7f2f0406f590 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 1 monclient: found mon.noname-a 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f0a9c9700 1 -- 172.21.15.104:0/4012735060 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0409a050 con 0x7f2f0406f960 2024-09-18T14:27:55.698 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.689+0000 7f2f091c6700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T14:27:55.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0a9c9700 1 -- 172.21.15.104:0/4012735060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f960 msgr2=0x7f2f0413eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0a9c9700 1 --2- 172.21.15.104:0/4012735060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f960 0x7f2f0413eb20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009f60 tx=0x7f2efc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf680d6532f1f36e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 1 -- 172.21.15.104:0/4012735060 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00e070 con 0x7f2f0406f960 2024-09-18T14:27:55.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: dump: 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.701 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 1 -- 172.21.15.104:0/4012735060 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc00be10 con 0x7f2f0406f960 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 5 monclient: authenticate success, global_id 15213 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 4 set_mon_vals no callback set 2024-09-18T14:27:55.702 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 1 -- 172.21.15.104:0/4012735060 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc0117d0 con 0x7f2f0406f960 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f091c6700 10 monclient: dump: 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.703 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 10 monclient: shutdown 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/4012735060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f960 msgr2=0x7f2f0413eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/4012735060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f960 0x7f2f0413eb20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009f60 tx=0x7f2efc009780 comp rx=0 tx=0).stop 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/4012735060 shutdown_connections 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/4012735060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f960 0x7f2f0413eb20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/4012735060 >> 172.21.15.104:0/4012735060 conn(0x7f2f040a11f0 msgr2=0x7f2f040a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/4012735060 shutdown_connections 2024-09-18T14:27:55.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.693+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/4012735060 wait complete. 2024-09-18T14:27:55.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient: build_initial_monmap 2024-09-18T14:27:55.708 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient: monmap: 2024-09-18T14:27:55.708 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:55.708 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 Processor -- start 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 -- start start 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient: init 2024-09-18T14:27:55.709 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 0x7f2f04145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): picked mon.noname-a con 0x7f2f0406f1c0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406f590 0x7f2f04093f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): picked mon.noname-b con 0x7f2f0406f590 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2f0406f960 0x7f2f04094460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): picked mon.noname-c con 0x7f2f0406f960 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0413e380 con 0x7f2f0406f1c0 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0409b3d0 con 0x7f2f0406f590 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:55.710 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f040a0210 con 0x7f2f0406f960 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0b1ca700 10 monclient(hunting): authenticate will time out at 1196.690767s 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 0x7f2f04145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 0x7f2f04145550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35744/0 (socket says 172.21.15.104:35744) 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2f0406f960 0x7f2f04094460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 -- 172.21.15.104:0/1776703693 learned_addr learned my addr 172.21.15.104:0/1776703693 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): get_auth_request con 0x7f2f0406f1c0 auth_method 0 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 10 monclient(hunting): get_auth_request con 0x7f2f0406f960 auth_method 0 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.711 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f099c7700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406f590 0x7f2f04093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a9c9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f099c7700 10 monclient(hunting): get_auth_request con 0x7f2f0406f590 auth_method 0 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f099c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f099c7700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f099c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient(hunting): handle_auth_done global_id 15219 payload 824 2024-09-18T14:27:55.712 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 -- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2f0406f960 msgr2=0x7f2f04094460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2f0406f960 0x7f2f04094460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 -- 172.21.15.104:0/1776703693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406f590 msgr2=0x7f2f04093f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406f590 0x7f2f04093f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient: _finish_hunting 0 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 monclient: found mon.noname-a 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.697+0000 7f2f0a1c8700 1 -- 172.21.15.104:0/1776703693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc009a20 con 0x7f2f0406f1c0 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0a1c8700 1 -- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 msgr2=0x7f2f04145550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0a1c8700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 0x7f2f04145550 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000c570 tx=0x7f2f00007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 1 -- 172.21.15.104:0/1776703693 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f000074f0 con 0x7f2f0406f1c0 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: dump: 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:55.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 1 -- 172.21.15.104:0/1776703693 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f00007670 con 0x7f2f0406f1c0 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 5 monclient: authenticate success, global_id 15219 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 10 monclient: _renew_subs 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0413ea40 con 0x7f2f0406f1c0 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 1 -- 172.21.15.104:0/1776703693 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00005a60 con 0x7f2f0406f1c0 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 10 monclient: dump: 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f089c5700 4 set_mon_vals no callback set 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 10 monclient: _renew_subs 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0413de70 con 0x7f2f0406f1c0 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:55.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f04063450 con 0x7f2f0406f1c0 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.701+0000 7f2efa7fc700 1 -- 172.21.15.104:0/1776703693 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f2f0001b3b0 con 0x7f2f0406f1c0 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.705+0000 7f2efa7fc700 1 --2- 172.21.15.104:0/1776703693 >> v2:172.21.15.174:6812/17603 conn(0x7f2ee8035a80 0x7f2ee8037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.705+0000 7f2efa7fc700 1 -- 172.21.15.104:0/1776703693 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2f000472b0 con 0x7f2f0406f1c0 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.705+0000 7f2f099c7700 1 --2- 172.21.15.104:0/1776703693 >> v2:172.21.15.174:6812/17603 conn(0x7f2ee8035a80 0x7f2ee8037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.705+0000 7f2f099c7700 10 monclient: get_auth_request con 0x7f2ee8035a80 auth_method 0 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.705+0000 7f2f099c7700 1 --2- 172.21.15.104:0/1776703693 >> v2:172.21.15.174:6812/17603 conn(0x7f2ee8035a80 0x7f2ee8037f30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40014a0 tx=0x7f2ef4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.709+0000 7f2efa7fc700 1 -- 172.21.15.104:0/1776703693 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2f000196e0 con 0x7f2f0406f1c0 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.709+0000 7f2efa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:55.718 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.709+0000 7f2efa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:55.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f040988d0 con 0x7f2ee8035a80 2024-09-18T14:27:55.882 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2efa7fc700 1 -- 172.21.15.104:0/1776703693 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2f040988d0 con 0x7f2ee8035a80 2024-09-18T14:27:55.885 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 >> v2:172.21.15.174:6812/17603 conn(0x7f2ee8035a80 msgr2=0x7f2ee8037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/1776703693 >> v2:172.21.15.174:6812/17603 conn(0x7f2ee8035a80 0x7f2ee8037f30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40014a0 tx=0x7f2ef4009040 comp rx=0 tx=0).stop 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2f0b1ca700 10 monclient: shutdown 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2f0b1ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 msgr2=0x7f2f04145550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.877+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 0x7f2f04145550 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000c570 tx=0x7f2f00007e60 comp rx=0 tx=0).stop 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 shutdown_connections 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/1776703693 >> v2:172.21.15.174:6812/17603 conn(0x7f2ee8035a80 0x7f2ee8037f30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f2f0406f960 0x7f2f04094460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406f590 0x7f2f04093f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 --2- 172.21.15.104:0/1776703693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f0406f1c0 0x7f2f04145550 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 >> 172.21.15.104:0/1776703693 conn(0x7f2f040a11f0 msgr2=0x7f2f0409ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:55.886 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 shutdown_connections 2024-09-18T14:27:55.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:55.881+0000 7f2f0b1ca700 1 -- 172.21.15.104:0/1776703693 wait complete. 2024-09-18T14:27:55.887 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T14:27:55.896 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T14:27:55.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:27:56.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient: get_monmap_and_config 2024-09-18T14:27:56.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient: monmap: 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:56.004985+0000 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:56.004985+0000 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:56.007 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 1 Processor -- start 2024-09-18T14:27:56.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 1 -- start start 2024-09-18T14:27:56.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient: init 2024-09-18T14:27:56.010 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 0x7f7094131be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient(hunting): picked mon.noname-b con 0x7f709409bc30 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7094132120 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7094132550 0x7f70940a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient(hunting): picked mon.noname-a con 0x7f7094132550 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709409c000 con 0x7f7094132120 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709409b530 con 0x7f7094132550 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709406f400 con 0x7f709409bc30 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.011 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709af75700 10 monclient(hunting): authenticate will time out at 1196.995923s 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f7099f73700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 0x7f7094131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.001+0000 7f709a774700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7094132550 0x7f70940a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7094132550 0x7f70940a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35750/0 (socket says 172.21.15.104:35750) 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 1 -- 172.21.15.104:0/1674587271 learned_addr learned my addr 172.21.15.104:0/1674587271 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 0x7f7094131be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43024/0 (socket says 172.21.15.104:43024) 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 10 monclient(hunting): get_auth_request con 0x7f7094132550 auth_method 0 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): get_auth_request con 0x7f709409bc30 auth_method 0 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.012 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709a774700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient(hunting): handle_auth_done global_id 34163 payload 274 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 -- 172.21.15.104:0/1674587271 >> v2:172.21.15.104:3300/0 conn(0x7f7094132550 msgr2=0x7f70940a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 --2- 172.21.15.104:0/1674587271 >> v2:172.21.15.104:3300/0 conn(0x7f7094132550 0x7f70940a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 -- 172.21.15.104:0/1674587271 >> v1:172.21.15.174:6789/0 conn(0x7f7094132120 legacy=0x7f70940a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 monclient: found mon.noname-b 2024-09-18T14:27:56.013 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.014 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 -- 172.21.15.104:0/1674587271 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709409a120 con 0x7f709409bc30 2024-09-18T14:27:56.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 -- 172.21.15.104:0/1674587271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 msgr2=0x7f7094131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7099f73700 1 --2- 172.21.15.104:0/1674587271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 0x7f7094131be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70840098e0 tx=0x7f70840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b58b3147b476de39 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.018 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 1 -- 172.21.15.104:0/1674587271 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400b850 con 0x7f709409bc30 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: dump: 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.019 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 5 monclient: authenticate success, global_id 34163 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 1 -- 172.21.15.104:0/1674587271 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708400b9b0 con 0x7f709409bc30 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 4 set_mon_vals no callback set 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 1 -- 172.21.15.104:0/1674587271 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400bd50 con 0x7f709409bc30 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f7098f71700 10 monclient: dump: 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.020 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 10 monclient: shutdown 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 1 -- 172.21.15.104:0/1674587271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 msgr2=0x7f7094131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.005+0000 7f709af75700 1 --2- 172.21.15.104:0/1674587271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 0x7f7094131be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70840098e0 tx=0x7f70840091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- 172.21.15.104:0/1674587271 shutdown_connections 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 --2- 172.21.15.104:0/1674587271 >> v2:172.21.15.104:3300/0 conn(0x7f7094132550 0x7f70940a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 --2- 172.21.15.104:0/1674587271 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f709409bc30 0x7f7094131be0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- 172.21.15.104:0/1674587271 >> 172.21.15.104:0/1674587271 conn(0x7f7094068120 msgr2=0x7f70940a53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- 172.21.15.104:0/1674587271 shutdown_connections 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- 172.21.15.104:0/1674587271 wait complete. 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient: monmap: 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:56.022 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 Processor -- start 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- start start 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient: init 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f709409bc30 0x7f70941410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): picked mon.noname-b con 0x7f709409bc30 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7094132120 0x7f7094131320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): picked mon.noname-a con 0x7f7094132120 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7094132550 0x7f709411bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): picked mon.noname-c con 0x7f7094132550 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709411f730 con 0x7f7094132120 2024-09-18T14:27:56.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709411f450 con 0x7f709409bc30 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709411fa10 con 0x7f7094132550 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709af75700 10 monclient(hunting): authenticate will time out at 1197.003455s 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7094132120 0x7f7094131320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7094132550 0x7f709411bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7094132120 0x7f7094131320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35756/0 (socket says 172.21.15.104:35756) 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099f73700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f709409bc30 0x7f70941410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 1 -- 172.21.15.104:0/4062186954 learned_addr learned my addr 172.21.15.104:0/4062186954 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7094132550 0x7f709411bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43028/0 (socket says 172.21.15.104:43028) 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 10 monclient(hunting): get_auth_request con 0x7f7094132120 auth_method 0 2024-09-18T14:27:56.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 10 monclient(hunting): get_auth_request con 0x7f7094132550 auth_method 0 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099f73700 10 monclient(hunting): get_auth_request con 0x7f709409bc30 auth_method 0 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099f73700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099f73700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099f73700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f7099772700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.009+0000 7f709a774700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709a774700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099f73700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099f73700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099f73700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 10 monclient(hunting): handle_auth_done global_id 15225 payload 824 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 -- 172.21.15.104:0/4062186954 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7094132550 msgr2=0x7f709411bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 --2- 172.21.15.104:0/4062186954 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7094132550 0x7f709411bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 -- 172.21.15.104:0/4062186954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f709409bc30 msgr2=0x7f70941410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 --2- 172.21.15.104:0/4062186954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f709409bc30 0x7f70941410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 monclient: found mon.noname-a 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 -- 172.21.15.104:0/4062186954 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7084009420 con 0x7f7094132120 2024-09-18T14:27:56.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709a774700 1 --2- 172.21.15.104:0/4062186954 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7094132550 0x7f709411bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:56.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 -- 172.21.15.104:0/4062186954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7094132120 msgr2=0x7f7094131320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7099772700 1 --2- 172.21.15.104:0/4062186954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7094132120 0x7f7094131320 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f709000c5d0 tx=0x7f7090007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70900117a0 con 0x7f7094132120 2024-09-18T14:27:56.029 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: dump: 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.030 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709af75700 5 monclient: authenticate success, global_id 15225 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090011900 con 0x7f7094132120 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709af75700 10 monclient: _renew_subs 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090011cc0 con 0x7f7094132120 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709af75700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7094141ea0 con 0x7f7094132120 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f708bfff700 4 set_mon_vals no callback set 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f7089ffb700 10 monclient: dump: 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.031 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709af75700 10 monclient: _renew_subs 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709af75700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.013+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7094141790 con 0x7f7094132120 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f709af75700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70940636d0 con 0x7f7094132120 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f7090012630 con 0x7f7094132120 2024-09-18T14:27:56.032 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7089ffb700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.174:6812/17603 conn(0x7f7078035a80 0x7f7078037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7090047b60 con 0x7f7094132120 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7089ffb700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.104:6800/17762 conn(0x7f707803b5a0 0x7f707803d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 --> v2:172.21.15.104:6800/17762 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f707803e020 con 0x7f707803b5a0 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=327) v2 ==== 24+0+0 (secure 0 0 0) 0x7f709004c2a0 con 0x7f7094132120 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7089ffb700 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f709a774700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.104:6800/17762 conn(0x7f707803b5a0 0x7f707803d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7099f73700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.174:6812/17603 conn(0x7f7078035a80 0x7f7078037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f709a774700 10 monclient: get_auth_request con 0x7f707803b5a0 auth_method 0 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.017+0000 7f7099f73700 10 monclient: get_auth_request con 0x7f7078035a80 auth_method 0 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.021+0000 7f7099f73700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.174:6812/17603 conn(0x7f7078035a80 0x7f7078037f30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f70840093f0 tx=0x7f7084016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.021+0000 7f709a774700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.104:6800/17762 conn(0x7f707803b5a0 0x7f707803d9b0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.021+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== osd.2 v2:172.21.15.104:6800/17762 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f707803e020 con 0x7f707803b5a0 2024-09-18T14:27:56.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.037+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 --> v2:172.21.15.104:6800/17762 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f709411e730 con 0x7f707803b5a0 2024-09-18T14:27:56.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f7089ffb700 1 -- 172.21.15.104:0/4062186954 <== osd.2 v2:172.21.15.104:6800/17762 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f709411e730 con 0x7f707803b5a0 2024-09-18T14:27:56.045 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:56.045 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:27:56.045 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:56.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 >> v2:172.21.15.104:6800/17762 conn(0x7f707803b5a0 msgr2=0x7f707803d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.104:6800/17762 conn(0x7f707803b5a0 0x7f707803d9b0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.046 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 >> v2:172.21.15.174:6812/17603 conn(0x7f7078035a80 msgr2=0x7f7078037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 --2- 172.21.15.104:0/4062186954 >> v2:172.21.15.174:6812/17603 conn(0x7f7078035a80 0x7f7078037f30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f70840093f0 tx=0x7f7084016000 comp rx=0 tx=0).stop 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 10 monclient: shutdown 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7094132120 msgr2=0x7f7094131320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 --2- 172.21.15.104:0/4062186954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7094132120 0x7f7094131320 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f709000c5d0 tx=0x7f7090007f60 comp rx=0 tx=0).stop 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709a774700 1 -- 172.21.15.104:0/4062186954 reap_dead start 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 shutdown_connections 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 >> 172.21.15.104:0/4062186954 conn(0x7f7094068120 msgr2=0x7f70940a0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 shutdown_connections 2024-09-18T14:27:56.047 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.041+0000 7f709af75700 1 -- 172.21.15.104:0/4062186954 wait complete. 2024-09-18T14:27:56.058 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T14:27:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient: get_monmap_and_config 2024-09-18T14:27:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient: monmap: 2024-09-18T14:27:56.165 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:56.164010+0000 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:56.164010+0000 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:56.166 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 1 Processor -- start 2024-09-18T14:27:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 1 -- start start 2024-09-18T14:27:56.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient: init 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbf1c09b750 0x7fbf1c099c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient(hunting): picked mon.noname-a con 0x7fbf1c09b750 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbf1c09bb20 0x7fbf1c09a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient(hunting): picked mon.noname-c con 0x7fbf1c09bb20 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf1c09cd00 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1c099290 con 0x7fbf1c09cd00 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1c09afe0 con 0x7fbf1c09b750 2024-09-18T14:27:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1c0997f0 con 0x7fbf1c09bb20 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf216c8700 10 monclient(hunting): authenticate will time out at 1197.155614s 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbf1c09b750 0x7fbf1c099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbf1c09b750 0x7fbf1c099c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35760/0 (socket says 172.21.15.104:35760) 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1affd700 1 -- 172.21.15.104:0/2577553856 learned_addr learned my addr 172.21.15.104:0/2577553856 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1affd700 10 monclient(hunting): get_auth_request con 0x7fbf1c09b750 auth_method 0 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.161+0000 7fbf1a7fc700 1 --2- 172.21.15.104:0/2577553856 >> v2:172.21.15.174:3300/0 conn(0x7fbf1c09bb20 0x7fbf1c09a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1a7fc700 10 monclient(hunting): get_auth_request con 0x7fbf1c09bb20 auth_method 0 2024-09-18T14:27:56.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 10 monclient(hunting): handle_auth_done global_id 15228 payload 274 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 1 -- 172.21.15.104:0/2577553856 >> v2:172.21.15.174:3300/0 conn(0x7fbf1c09bb20 msgr2=0x7fbf1c09a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 1 --2- 172.21.15.104:0/2577553856 >> v2:172.21.15.174:3300/0 conn(0x7fbf1c09bb20 0x7fbf1c09a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 1 -- 172.21.15.104:0/2577553856 >> v1:172.21.15.174:6789/0 conn(0x7fbf1c09cd00 legacy=0x7fbf1c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.172 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 1 monclient: found mon.noname-a 2024-09-18T14:27:56.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 1 -- 172.21.15.104:0/2577553856 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c09ea60 con 0x7fbf1c09b750 2024-09-18T14:27:56.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 1 -- 172.21.15.104:0/2577553856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09b750 msgr2=0x7fbf1c099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf1affd700 1 --2- 172.21.15.104:0/2577553856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09b750 0x7fbf1c099c40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c064960 tx=0x7fbf14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d1e6f60bdf6eda server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 1 -- 172.21.15.104:0/2577553856 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1400e070 con 0x7fbf1c09b750 2024-09-18T14:27:56.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: dump: 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf216c8700 5 monclient: authenticate success, global_id 15228 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf216c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 1 -- 172.21.15.104:0/2577553856 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1400bc30 con 0x7fbf1c09b750 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf216c8700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 1 -- 172.21.15.104:0/2577553856 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf140092e0 con 0x7fbf1c09b750 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf216c8700 4 set_mon_vals no callback set 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.165+0000 7fbf19ffb700 10 monclient: dump: 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.176 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 10 monclient: shutdown 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 -- 172.21.15.104:0/2577553856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09b750 msgr2=0x7fbf1c099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 --2- 172.21.15.104:0/2577553856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09b750 0x7fbf1c099c40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c064960 tx=0x7fbf14009840 comp rx=0 tx=0).stop 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 -- 172.21.15.104:0/2577553856 shutdown_connections 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 --2- 172.21.15.104:0/2577553856 >> v2:172.21.15.174:3300/0 conn(0x7fbf1c09bb20 0x7fbf1c09a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 --2- 172.21.15.104:0/2577553856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09b750 0x7fbf1c099c40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 -- 172.21.15.104:0/2577553856 >> 172.21.15.104:0/2577553856 conn(0x7fbf1c069180 msgr2=0x7fbf1c0a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 -- 172.21.15.104:0/2577553856 shutdown_connections 2024-09-18T14:27:56.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 -- 172.21.15.104:0/2577553856 wait complete. 2024-09-18T14:27:56.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 10 monclient: monmap: 2024-09-18T14:27:56.181 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.181 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:56.181 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 Processor -- start 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 1 -- start start 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.169+0000 7fbf216c8700 10 monclient: init 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf1c09b750 0x7fbf1c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): picked mon.noname-b con 0x7fbf1c09b750 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf1c09bb20 0x7fbf1c130000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): picked mon.noname-c con 0x7fbf1c09bb20 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 0x7fbf1c132d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): picked mon.noname-a con 0x7fbf1c09cd00 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c09c030 con 0x7fbf1c09cd00 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c136530 con 0x7fbf1c09b750 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c130af0 con 0x7fbf1c09bb20 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient(hunting): authenticate will time out at 1197.164366s 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 0x7fbf1c132d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf1c09bb20 0x7fbf1c130000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 0x7fbf1c132d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35774/0 (socket says 172.21.15.104:35774) 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/557177033 learned_addr learned my addr 172.21.15.104:0/557177033 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): get_auth_request con 0x7fbf1c09cd00 auth_method 0 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 10 monclient(hunting): get_auth_request con 0x7fbf1c09bb20 auth_method 0 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf1c09b750 0x7fbf1c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 10 monclient(hunting): get_auth_request con 0x7fbf1c09b750 auth_method 0 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient(hunting): handle_auth_done global_id 15231 payload 824 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf1c09bb20 msgr2=0x7fbf1c130000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf1c09bb20 0x7fbf1c130000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/557177033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf1c09b750 msgr2=0x7fbf1c13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf1c09b750 0x7fbf1c13f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 monclient: found mon.noname-a 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/557177033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf14009ae0 con 0x7fbf1c09cd00 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 -- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 msgr2=0x7fbf1c132d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf1b7fe700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 0x7fbf1c132d60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00c5b0 tx=0x7fbf0c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 1 -- 172.21.15.104:0/557177033 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c0117a0 con 0x7fbf1c09cd00 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: dump: 2024-09-18T14:27:56.186 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 5 monclient: authenticate success, global_id 15231 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient: _renew_subs 2024-09-18T14:27:56.187 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 1 -- 172.21.15.104:0/557177033 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c011900 con 0x7fbf1c09cd00 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1c136810 con 0x7fbf1c09cd00 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 1 -- 172.21.15.104:0/557177033 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c011cc0 con 0x7fbf1c09cd00 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf02ffd700 10 monclient: dump: 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.188 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf197fa700 4 set_mon_vals no callback set 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient: _renew_subs 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.189 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.173+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c13fdd0 con 0x7fbf1c09cd00 2024-09-18T14:27:56.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf02ffd700 1 -- 172.21.15.104:0/557177033 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fbf0c012630 con 0x7fbf1c09cd00 2024-09-18T14:27:56.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf216c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:56.190 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf216c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf216c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1c0636d0 con 0x7fbf1c09cd00 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf02ffd700 1 --2- 172.21.15.104:0/557177033 >> v2:172.21.15.174:6812/17603 conn(0x7fbf08035ab0 0x7fbf08037f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf02ffd700 1 -- 172.21.15.104:0/557177033 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbf0c047a20 con 0x7fbf1c09cd00 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf1affd700 1 --2- 172.21.15.104:0/557177033 >> v2:172.21.15.174:6812/17603 conn(0x7fbf08035ab0 0x7fbf08037f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.177+0000 7fbf1affd700 10 monclient: get_auth_request con 0x7fbf08035ab0 auth_method 0 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.181+0000 7fbf1affd700 1 --2- 172.21.15.104:0/557177033 >> v2:172.21.15.174:6812/17603 conn(0x7fbf08035ab0 0x7fbf08037f60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009a60 tx=0x7fbf14017000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.181+0000 7fbf02ffd700 1 -- 172.21.15.104:0/557177033 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbf0c011e20 con 0x7fbf1c09cd00 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.181+0000 7fbf02ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:56.191 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.181+0000 7fbf02ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:56.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf1c13f760 con 0x7fbf08035ab0 2024-09-18T14:27:56.354 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf02ffd700 1 -- 172.21.15.104:0/557177033 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbf1c13f760 con 0x7fbf08035ab0 2024-09-18T14:27:56.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 >> v2:172.21.15.174:6812/17603 conn(0x7fbf08035ab0 msgr2=0x7fbf08037f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 --2- 172.21.15.104:0/557177033 >> v2:172.21.15.174:6812/17603 conn(0x7fbf08035ab0 0x7fbf08037f60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009a60 tx=0x7fbf14017000 comp rx=0 tx=0).stop 2024-09-18T14:27:56.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 10 monclient: shutdown 2024-09-18T14:27:56.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 msgr2=0x7fbf1c132d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.357 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 0x7fbf1c132d60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00c5b0 tx=0x7fbf0c007f60 comp rx=0 tx=0).stop 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 shutdown_connections 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 --2- 172.21.15.104:0/557177033 >> v2:172.21.15.174:6812/17603 conn(0x7fbf08035ab0 0x7fbf08037f60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.349+0000 7fbf216c8700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf1c09cd00 0x7fbf1c132d60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.353+0000 7fbf216c8700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fbf1c09bb20 0x7fbf1c130000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.353+0000 7fbf216c8700 1 --2- 172.21.15.104:0/557177033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf1c09b750 0x7fbf1c13f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.353+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 >> 172.21.15.104:0/557177033 conn(0x7fbf1c069180 msgr2=0x7fbf1c097c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.353+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 shutdown_connections 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.353+0000 7fbf216c8700 1 -- 172.21.15.104:0/557177033 wait complete. 2024-09-18T14:27:56.358 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T14:27:56.368 INFO:tasks.ceph:Scrubbing osd.3 2024-09-18T14:27:56.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:27:56.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.473+0000 7fb656a51700 10 monclient: get_monmap_and_config 2024-09-18T14:27:56.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.473+0000 7fb656a51700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.473+0000 7fb656a51700 10 monclient: monmap: 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:56.478198+0000 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:56.478198+0000 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:56.480 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.473+0000 7fb656a51700 1 Processor -- start 2024-09-18T14:27:56.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.473+0000 7fb656a51700 1 -- start start 2024-09-18T14:27:56.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.473+0000 7fb656a51700 10 monclient: init 2024-09-18T14:27:56.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T14:27:56.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb650095c80 0x7fb65006a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient(hunting): picked mon.noname-c con 0x7fb650095c80 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:56.484 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb650096050 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb650138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient(hunting): picked mon.noname-b con 0x7fb650096b60 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65009f230 con 0x7fb650096050 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65009f7c0 con 0x7fb650095c80 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65009e100 con 0x7fb650096b60 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656a51700 10 monclient(hunting): authenticate will time out at 1197.469096s 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb650138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb650138fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43054/0 (socket says 172.21.15.104:43054) 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 -- 172.21.15.104:0/2359948786 learned_addr learned my addr 172.21.15.104:0/2359948786 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.485 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): get_auth_request con 0x7fb650096b60 auth_method 0 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 1 --2- 172.21.15.104:0/2359948786 >> v2:172.21.15.174:3300/0 conn(0x7fb650095c80 0x7fb65006a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 10 monclient(hunting): get_auth_request con 0x7fb650095c80 auth_method 0 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.486 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb655a4f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient(hunting): handle_auth_done global_id 34172 payload 274 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 -- 172.21.15.104:0/2359948786 >> v2:172.21.15.174:3300/0 conn(0x7fb650095c80 msgr2=0x7fb65006a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 --2- 172.21.15.104:0/2359948786 >> v2:172.21.15.174:3300/0 conn(0x7fb650095c80 0x7fb65006a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 -- 172.21.15.104:0/2359948786 >> v1:172.21.15.174:6789/0 conn(0x7fb650096050 legacy=0x7fb650096450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 monclient: found mon.noname-b 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.487 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.477+0000 7fb656250700 1 -- 172.21.15.104:0/2359948786 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65009f4e0 con 0x7fb650096b60 2024-09-18T14:27:56.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 1 -- 172.21.15.104:0/2359948786 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 msgr2=0x7fb650138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 1 --2- 172.21.15.104:0/2359948786 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb650138fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb648009f20 tx=0x7fb648009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=586747826773d480 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 1 -- 172.21.15.104:0/2359948786 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800e070 con 0x7fb650096b60 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: dump: 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.492 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 1 -- 172.21.15.104:0/2359948786 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64800bbe0 con 0x7fb650096b60 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 5 monclient: authenticate success, global_id 34172 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 4 set_mon_vals no callback set 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 1 -- 172.21.15.104:0/2359948786 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6480092e0 con 0x7fb650096b60 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb654a4d700 10 monclient: dump: 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient: shutdown 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- 172.21.15.104:0/2359948786 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 msgr2=0x7fb650138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 --2- 172.21.15.104:0/2359948786 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb650138fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb648009f20 tx=0x7fb648009840 comp rx=0 tx=0).stop 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- 172.21.15.104:0/2359948786 shutdown_connections 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 --2- 172.21.15.104:0/2359948786 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb650138fa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 --2- 172.21.15.104:0/2359948786 >> v2:172.21.15.174:3300/0 conn(0x7fb650095c80 0x7fb65006a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- 172.21.15.104:0/2359948786 >> 172.21.15.104:0/2359948786 conn(0x7fb650068160 msgr2=0x7fb6500a33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- 172.21.15.104:0/2359948786 shutdown_connections 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- 172.21.15.104:0/2359948786 wait complete. 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient: monmap: 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:56.495 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 Processor -- start 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- start start 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient: init 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb650095c80 0x7fb650138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): picked mon.noname-a con 0x7fb650095c80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:56.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb650096050 0x7fb65011e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): picked mon.noname-b con 0x7fb650096050 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb65011ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): picked mon.noname-c con 0x7fb650096b60 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65006f440 con 0x7fb650095c80 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65006e760 con 0x7fb650096050 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb650122a50 con 0x7fb650096b60 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656a51700 10 monclient(hunting): authenticate will time out at 1197.475344s 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb650095c80 0x7fb650138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb65011ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb650095c80 0x7fb650138030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35788/0 (socket says 172.21.15.104:35788) 2024-09-18T14:27:56.497 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 1 -- 172.21.15.104:0/543859472 learned_addr learned my addr 172.21.15.104:0/543859472 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 10 monclient(hunting): get_auth_request con 0x7fb650095c80 auth_method 0 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 10 monclient(hunting): get_auth_request con 0x7fb650096b60 auth_method 0 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb65524e700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb650096050 0x7fb65011e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb655a4f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb65524e700 10 monclient(hunting): get_auth_request con 0x7fb650096050 auth_method 0 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb65524e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb65524e700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb65524e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.481+0000 7fb656250700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb65524e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb65524e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb65524e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 10 monclient(hunting): handle_auth_done global_id 34175 payload 824 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 -- 172.21.15.104:0/543859472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb650096050 msgr2=0x7fb65011e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb650096050 0x7fb65011e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 -- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb650095c80 msgr2=0x7fb650138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb650095c80 0x7fb650138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 monclient: found mon.noname-c 2024-09-18T14:27:56.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 -- 172.21.15.104:0/543859472 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb648009ae0 con 0x7fb650096b60 2024-09-18T14:27:56.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb655a4f700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb650095c80 0x7fb650138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:56.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb65524e700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb650096050 0x7fb65011e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:56.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 -- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 msgr2=0x7fb65011ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.500 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656250700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb65011ecb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb648003000 tx=0x7fb648017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648012a60 con 0x7fb650096b60 2024-09-18T14:27:56.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.502 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: dump: 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.503 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656a51700 5 monclient: authenticate success, global_id 34175 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb648012be0 con 0x7fb650096b60 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656a51700 10 monclient: _renew_subs 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656a51700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb650120770 con 0x7fb650096b60 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64801d420 con 0x7fb650096b60 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb645ffb700 10 monclient: dump: 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb647fff700 4 set_mon_vals no callback set 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656a51700 10 monclient: _renew_subs 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656a51700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.485+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb650120260 con 0x7fb650096b60 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb656a51700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb650064b30 con 0x7fb650096b60 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb656250700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb65011ecb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb648003000 tx=0x7fb648017000 comp rx=0 tx=0).queue_received will delay after 1.72667e+09 on 0x7fb648049420 osd_map(327..327 src has 124..327) v4 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7fb64801d7d0 con 0x7fb650096b60 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb645ffb700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6812/17603 conn(0x7fb634035a80 0x7fb634037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb648049420 con 0x7fb650096b60 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb645ffb700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6808/17845 conn(0x7fb63403b520 0x7fb63403d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 --> v2:172.21.15.174:6808/17845 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb63403dfa0 con 0x7fb63403b520 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb655a4f700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6812/17603 conn(0x7fb634035a80 0x7fb634037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb655a4f700 10 monclient: get_auth_request con 0x7fb634035a80 auth_method 0 2024-09-18T14:27:56.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb65524e700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6808/17845 conn(0x7fb63403b520 0x7fb63403d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.489+0000 7fb65524e700 10 monclient: get_auth_request con 0x7fb63403b520 auth_method 0 2024-09-18T14:27:56.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.493+0000 7fb655a4f700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6812/17603 conn(0x7fb634035a80 0x7fb634037f30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009fd0 tx=0x7fb64c0091a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.493+0000 7fb65524e700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6808/17845 conn(0x7fb63403b520 0x7fb63403d930 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.497+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== osd.3 v2:172.21.15.174:6808/17845 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fb63403dfa0 con 0x7fb63403b520 2024-09-18T14:27:56.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 --> v2:172.21.15.174:6808/17845 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb650120e30 con 0x7fb63403b520 2024-09-18T14:27:56.519 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== osd.3 v2:172.21.15.174:6808/17845 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb650120e30 con 0x7fb63403b520 2024-09-18T14:27:56.519 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:56.519 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:27:56.519 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:56.520 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6808/17845 conn(0x7fb63403b520 msgr2=0x7fb63403d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6808/17845 conn(0x7fb63403b520 0x7fb63403d930 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6812/17603 conn(0x7fb634035a80 msgr2=0x7fb634037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 --2- 172.21.15.104:0/543859472 >> v2:172.21.15.174:6812/17603 conn(0x7fb634035a80 0x7fb634037f30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009fd0 tx=0x7fb64c0091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 10 monclient: shutdown 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 20 monclient: shutdown canceling and discarding version request 1 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 msgr2=0x7fb65011ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 --2- 172.21.15.104:0/543859472 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7fb650096b60 0x7fb65011ecb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb648003000 tx=0x7fb648017000 comp rx=0 tx=0).stop 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656250700 1 -- 172.21.15.104:0/543859472 reap_dead start 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb645ffb700 1 -- 172.21.15.104:0/543859472 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_get_version_reply(handle=1 version=327) v2 ==== 24+0+0 (unknown 0 0 0) 0x7fb64804b990 con 0x7fb650096b60 2024-09-18T14:27:56.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb645ffb700 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=327) v2 2024-09-18T14:27:56.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 shutdown_connections 2024-09-18T14:27:56.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 >> 172.21.15.104:0/543859472 conn(0x7fb650068160 msgr2=0x7fb6500a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 shutdown_connections 2024-09-18T14:27:56.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.513+0000 7fb656a51700 1 -- 172.21.15.104:0/543859472 wait complete. 2024-09-18T14:27:56.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient: get_monmap_and_config 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient: monmap: 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:56.640396+0000 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:56.640396+0000 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:56.642 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:56.643 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:56.643 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:56.643 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:56.643 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 1 Processor -- start 2024-09-18T14:27:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 1 -- start start 2024-09-18T14:27:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient: init 2024-09-18T14:27:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T14:27:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0efc09ed50 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:56.645 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0efc096160 0x7f0efc137510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient(hunting): picked mon.noname-a con 0x7f0efc096160 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc096560 0x7f0efc139a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient(hunting): picked mon.noname-b con 0x7f0efc096560 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0efc06c630 con 0x7f0efc09ed50 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0efc06d6f0 con 0x7f0efc096160 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc09e1b0 con 0x7f0efc096560 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f03cf8700 10 monclient(hunting): authenticate will time out at 1197.630780s 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0efc096160 0x7f0efc137510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0efc096160 0x7f0efc137510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35798/0 (socket says 172.21.15.104:35798) 2024-09-18T14:27:56.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 1 -- 172.21.15.104:0/259839994 learned_addr learned my addr 172.21.15.104:0/259839994 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc096560 0x7f0efc139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 10 monclient(hunting): get_auth_request con 0x7f0efc096160 auth_method 0 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): get_auth_request con 0x7f0efc096560 auth_method 0 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.647 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient(hunting): handle_auth_done global_id 34178 payload 274 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 1 -- 172.21.15.104:0/259839994 >> v2:172.21.15.104:3300/0 conn(0x7f0efc096160 msgr2=0x7f0efc137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 1 --2- 172.21.15.104:0/259839994 >> v2:172.21.15.104:3300/0 conn(0x7f0efc096160 0x7f0efc137510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 1 -- 172.21.15.104:0/259839994 >> v1:172.21.15.174:6789/0 conn(0x7f0efc09ed50 legacy=0x7f0efc095a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 1 monclient: found mon.noname-b 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f02295700 1 -- 172.21.15.104:0/259839994 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc096f50 con 0x7f0efc096560 2024-09-18T14:27:56.648 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.637+0000 7f0f01293700 1 --2- 172.21.15.104:0/259839994 >> v2:172.21.15.104:3300/0 conn(0x7f0efc096160 0x7f0efc137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f02295700 1 -- 172.21.15.104:0/259839994 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc096560 msgr2=0x7f0efc139a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f02295700 1 --2- 172.21.15.104:0/259839994 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc096560 0x7f0efc139a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009e60 tx=0x7f0ef4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ffa16ec18182d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 1 -- 172.21.15.104:0/259839994 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400e070 con 0x7f0efc096560 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: dump: 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.650 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 1 -- 172.21.15.104:0/259839994 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef400bd70 con 0x7f0efc096560 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 5 monclient: authenticate success, global_id 34178 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:56.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 1 -- 172.21.15.104:0/259839994 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4012680 con 0x7f0efc096560 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 4 set_mon_vals no callback set 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f00a92700 10 monclient: dump: 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.652 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 10 monclient: shutdown 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 -- 172.21.15.104:0/259839994 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc096560 msgr2=0x7f0efc139a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/259839994 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc096560 0x7f0efc139a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009e60 tx=0x7f0ef4009780 comp rx=0 tx=0).stop 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 -- 172.21.15.104:0/259839994 shutdown_connections 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/259839994 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc096560 0x7f0efc139a60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/259839994 >> v2:172.21.15.104:3300/0 conn(0x7f0efc096160 0x7f0efc137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 -- 172.21.15.104:0/259839994 >> 172.21.15.104:0/259839994 conn(0x7f0efc0a3510 msgr2=0x7f0efc0a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 -- 172.21.15.104:0/259839994 shutdown_connections 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.641+0000 7f0f03cf8700 1 -- 172.21.15.104:0/259839994 wait complete. 2024-09-18T14:27:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient: monmap: 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 Processor -- start 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 -- start start 2024-09-18T14:27:56.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient: init 2024-09-18T14:27:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T14:27:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 0x7f0efc13ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): picked mon.noname-a con 0x7f0efc096160 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0efc096560 0x7f0efc13d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): picked mon.noname-b con 0x7f0efc096560 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc09ed50 0x7f0efc141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): picked mon.noname-c con 0x7f0efc09ed50 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc122da0 con 0x7f0efc096160 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc122ac0 con 0x7f0efc096560 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc1202c0 con 0x7f0efc09ed50 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f03cf8700 10 monclient(hunting): authenticate will time out at 1197.638042s 2024-09-18T14:27:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 0x7f0efc13ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc09ed50 0x7f0efc141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 0x7f0efc13ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35808/0 (socket says 172.21.15.104:35808) 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 -- 172.21.15.104:0/2099529246 learned_addr learned my addr 172.21.15.104:0/2099529246 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): get_auth_request con 0x7f0efc096160 auth_method 0 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 10 monclient(hunting): get_auth_request con 0x7f0efc09ed50 auth_method 0 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f02295700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01293700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0efc096560 0x7f0efc13d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01293700 10 monclient(hunting): get_auth_request con 0x7f0efc096560 auth_method 0 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01293700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01293700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01293700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient(hunting): handle_auth_done global_id 15240 payload 824 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 -- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc09ed50 msgr2=0x7f0efc141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc09ed50 0x7f0efc141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 -- 172.21.15.104:0/2099529246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0efc096560 msgr2=0x7f0efc13d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0efc096560 0x7f0efc13d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 monclient: found mon.noname-a 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.645+0000 7f0f01a94700 1 -- 172.21.15.104:0/2099529246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec0094f0 con 0x7f0efc096160 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f01a94700 1 -- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 msgr2=0x7f0efc13ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f01a94700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 0x7f0efc13ddb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c550 tx=0x7f0ef8007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 1 -- 172.21.15.104:0/2099529246 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80117a0 con 0x7f0efc096160 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: dump: 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 5 monclient: authenticate success, global_id 15240 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 10 monclient: _renew_subs 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 1 -- 172.21.15.104:0/2099529246 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8011900 con 0x7f0efc096160 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef4009a20 con 0x7f0efc096160 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 1 -- 172.21.15.104:0/2099529246 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8011cc0 con 0x7f0efc096160 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef3fff700 4 set_mon_vals no callback set 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 10 monclient: dump: 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.664 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 10 monclient: _renew_subs 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc13d9a0 con 0x7f0efc096160 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 1 -- 172.21.15.104:0/2099529246 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f0ef8012630 con 0x7f0efc096160 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 1 --2- 172.21.15.104:0/2099529246 >> v2:172.21.15.174:6812/17603 conn(0x7f0ee0035ab0 0x7f0ee0037f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0ef1ffb700 1 -- 172.21.15.104:0/2099529246 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0ef8047bb0 con 0x7f0efc096160 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f01293700 1 --2- 172.21.15.104:0/2099529246 >> v2:172.21.15.174:6812/17603 conn(0x7f0ee0035ab0 0x7f0ee0037f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc122610 con 0x7f0efc096160 2024-09-18T14:27:56.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.649+0000 7f0f01293700 10 monclient: get_auth_request con 0x7f0ee0035ab0 auth_method 0 2024-09-18T14:27:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.653+0000 7f0f01293700 1 --2- 172.21.15.104:0/2099529246 >> v2:172.21.15.174:6812/17603 conn(0x7f0ee0035ab0 0x7f0ee0037f60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0efc06f3e0 tx=0x7f0eec008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.661+0000 7f0ef1ffb700 1 -- 172.21.15.104:0/2099529246 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0ef80108c0 con 0x7f0efc096160 2024-09-18T14:27:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.661+0000 7f0ef1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.661+0000 7f0ef1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:56.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.821+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0efc13dbc0 con 0x7f0ee0035ab0 2024-09-18T14:27:56.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0ef1ffb700 1 -- 172.21.15.104:0/2099529246 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0eec011a20 con 0x7f0ee0035ab0 2024-09-18T14:27:56.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 >> v2:172.21.15.174:6812/17603 conn(0x7f0ee0035ab0 msgr2=0x7f0ee0037f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/2099529246 >> v2:172.21.15.174:6812/17603 conn(0x7f0ee0035ab0 0x7f0ee0037f60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0efc06f3e0 tx=0x7f0eec008040 comp rx=0 tx=0).stop 2024-09-18T14:27:56.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 10 monclient: shutdown 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 msgr2=0x7f0efc13ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 0x7f0efc13ddb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c550 tx=0x7f0ef8007f60 comp rx=0 tx=0).stop 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 shutdown_connections 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/2099529246 >> v2:172.21.15.174:6812/17603 conn(0x7f0ee0035ab0 0x7f0ee0037f60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f0efc09ed50 0x7f0efc141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0efc096560 0x7f0efc13d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 --2- 172.21.15.104:0/2099529246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0efc096160 0x7f0efc13ddb0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 >> 172.21.15.104:0/2099529246 conn(0x7f0efc0a3510 msgr2=0x7f0efc09c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 shutdown_connections 2024-09-18T14:27:56.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.825+0000 7f0f03cf8700 1 -- 172.21.15.104:0/2099529246 wait complete. 2024-09-18T14:27:56.833 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 3 to deep-scrub 2024-09-18T14:27:56.843 INFO:tasks.ceph:Scrubbing osd.4 2024-09-18T14:27:56.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient: get_monmap_and_config 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient: monmap: 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:56.956310+0000 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:56.956310+0000 2024-09-18T14:27:56.958 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.959 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.959 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:56.959 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:56.959 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:56.959 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:56.959 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:56.959 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 1 Processor -- start 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 1 -- start start 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient: init 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1c8c09bc30 0x7f1c8c131be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1c8c09bc30 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c8c132120 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c0a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c132580 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c09c000 con 0x7f1c8c132120 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c09b530 con 0x7f1c8c09bc30 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c06f400 con 0x7f1c8c132580 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8b59e700 10 monclient(hunting): authenticate will time out at 1197.946777s 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c0a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c0a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43088/0 (socket says 172.21.15.104:43088) 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 -- 172.21.15.104:0/872306064 learned_addr learned my addr 172.21.15.104:0/872306064 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): get_auth_request con 0x7f1c8c132580 auth_method 0 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8a59c700 1 --2- 172.21.15.104:0/872306064 >> v2:172.21.15.174:3300/0 conn(0x7f1c8c09bc30 0x7f1c8c131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.963 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8a59c700 10 monclient(hunting): get_auth_request con 0x7f1c8c09bc30 auth_method 0 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient(hunting): handle_auth_done global_id 34184 payload 274 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 -- 172.21.15.104:0/872306064 >> v2:172.21.15.174:3300/0 conn(0x7f1c8c09bc30 msgr2=0x7f1c8c131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 --2- 172.21.15.104:0/872306064 >> v2:172.21.15.174:3300/0 conn(0x7f1c8c09bc30 0x7f1c8c131be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 -- 172.21.15.104:0/872306064 >> v1:172.21.15.174:6789/0 conn(0x7f1c8c132120 legacy=0x7f1c8c0a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 monclient: found mon.noname-b 2024-09-18T14:27:56.964 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:56.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8ad9d700 1 -- 172.21.15.104:0/872306064 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c09a120 con 0x7f1c8c132580 2024-09-18T14:27:56.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.953+0000 7f1c8a59c700 1 --2- 172.21.15.104:0/872306064 >> v2:172.21.15.174:3300/0 conn(0x7f1c8c09bc30 0x7f1c8c131be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:27:56.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8ad9d700 1 -- 172.21.15.104:0/872306064 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 msgr2=0x7f1c8c0a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8ad9d700 1 --2- 172.21.15.104:0/872306064 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c0a0980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009f20 tx=0x7f1c84009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce4fdc94811411a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 1 -- 172.21.15.104:0/872306064 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8400e070 con 0x7f1c8c132580 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: dump: 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.970 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 1 -- 172.21.15.104:0/872306064 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8400bc90 con 0x7f1c8c132580 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 5 monclient: authenticate success, global_id 34184 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 1 -- 172.21.15.104:0/872306064 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840092e0 con 0x7f1c8c132580 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 4 set_mon_vals no callback set 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8959a700 10 monclient: dump: 2024-09-18T14:27:56.971 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 10 monclient: shutdown 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 -- 172.21.15.104:0/872306064 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 msgr2=0x7f1c8c0a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 --2- 172.21.15.104:0/872306064 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c0a0980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009f20 tx=0x7f1c84009840 comp rx=0 tx=0).stop 2024-09-18T14:27:56.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 -- 172.21.15.104:0/872306064 shutdown_connections 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 --2- 172.21.15.104:0/872306064 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c0a0980 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 --2- 172.21.15.104:0/872306064 >> v2:172.21.15.174:3300/0 conn(0x7f1c8c09bc30 0x7f1c8c131be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 -- 172.21.15.104:0/872306064 >> 172.21.15.104:0/872306064 conn(0x7f1c8c068120 msgr2=0x7f1c8c0a53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 -- 172.21.15.104:0/872306064 shutdown_connections 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.957+0000 7f1c8b59e700 1 -- 172.21.15.104:0/872306064 wait complete. 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient: build_initial_monmap 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient: monmap: 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.973 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 Processor -- start 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 -- start start 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient: init 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8c09bc30 0x7f1c8c140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c09bc30 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1c8c132120 0x7f1c8c130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f1c8c132120 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c11daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1c8c132580 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c06f630 con 0x7f1c8c132120 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c131d60 con 0x7f1c8c09bc30 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c141e20 con 0x7f1c8c132580 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8b59e700 10 monclient(hunting): authenticate will time out at 1197.954484s 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c11daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1c8c132120 0x7f1c8c130fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c11daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43104/0 (socket says 172.21.15.104:43104) 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8c09bc30 0x7f1c8c140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 1 -- 172.21.15.104:0/16684548 learned_addr learned my addr 172.21.15.104:0/16684548 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 10 monclient(hunting): get_auth_request con 0x7f1c8c132580 auth_method 0 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1c8c132120 0x7f1c8c130fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35818/0 (socket says 172.21.15.104:35818) 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): get_auth_request con 0x7f1c8c132120 auth_method 0 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 10 monclient(hunting): get_auth_request con 0x7f1c8c09bc30 auth_method 0 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient(hunting): handle_auth_done global_id 15243 payload 824 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 -- 172.21.15.104:0/16684548 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 msgr2=0x7f1c8c11daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 --2- 172.21.15.104:0/16684548 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c11daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 -- 172.21.15.104:0/16684548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8c09bc30 msgr2=0x7f1c8c140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 --2- 172.21.15.104:0/16684548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8c09bc30 0x7f1c8c140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient: _finish_hunting 0 2024-09-18T14:27:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 monclient: found mon.noname-a 2024-09-18T14:27:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c89d9b700 1 -- 172.21.15.104:0/16684548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c84009ae0 con 0x7f1c8c132120 2024-09-18T14:27:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.961+0000 7f1c8ad9d700 1 --2- 172.21.15.104:0/16684548 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f1c8c132580 0x7f1c8c11daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c89d9b700 1 -- 172.21.15.104:0/16684548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1c8c132120 msgr2=0x7f1c8c130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c89d9b700 1 --2- 172.21.15.104:0/16684548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1c8c132120 0x7f1c8c130fc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400c4e0 tx=0x7f1c74007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7400ce90 con 0x7f1c8c132120 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: dump: 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: _finish_auth 0 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c740117a0 con 0x7f1c8c132120 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 5 monclient: authenticate success, global_id 15243 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 10 monclient: _renew_subs 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c11f5e0 con 0x7f1c8c132120 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74011b40 con 0x7f1c8c132120 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 10 monclient: dump: 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c88d99700 4 set_mon_vals no callback set 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 10 monclient: _renew_subs 2024-09-18T14:27:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c1211c0 con 0x7f1c8c132120 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c8c0636d0 con 0x7f1c8c132120 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.965+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f1c7401c460 con 0x7f1c8c132120 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c827fc700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6812/17603 conn(0x7f1c6c035a80 0x7f1c6c037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1c74047ac0 con 0x7f1c8c132120 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c827fc700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6800/17847 conn(0x7f1c6c03b430 0x7f1c6c03d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 --> v2:172.21.15.174:6800/17847 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c6c03deb0 con 0x7f1c6c03b430 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=327) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c7404c2a0 con 0x7f1c8c132120 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c827fc700 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c8a59c700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6812/17603 conn(0x7f1c6c035a80 0x7f1c6c037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c8a59c700 10 monclient: get_auth_request con 0x7f1c6c035a80 auth_method 0 2024-09-18T14:27:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c8ad9d700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6800/17847 conn(0x7f1c6c03b430 0x7f1c6c03d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c8ad9d700 10 monclient: get_auth_request con 0x7f1c6c03b430 auth_method 0 2024-09-18T14:27:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.969+0000 7f1c8a59c700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6812/17603 conn(0x7f1c6c035a80 0x7f1c6c037f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c006f00 tx=0x7f1c7c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.973+0000 7f1c8ad9d700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6800/17847 conn(0x7f1c6c03b430 0x7f1c6c03d840 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.973+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== osd.4 v2:172.21.15.174:6800/17847 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f1c6c03deb0 con 0x7f1c6c03b430 2024-09-18T14:27:56.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 --> v2:172.21.15.174:6800/17847 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1c8c141760 con 0x7f1c6c03b430 2024-09-18T14:27:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c827fc700 1 -- 172.21.15.104:0/16684548 <== osd.4 v2:172.21.15.174:6800/17847 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1c8c141760 con 0x7f1c6c03b430 2024-09-18T14:27:57.000 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:57.000 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:27:57.000 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6800/17847 conn(0x7f1c6c03b430 msgr2=0x7f1c6c03d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6800/17847 conn(0x7f1c6c03b430 0x7f1c6c03d840 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6812/17603 conn(0x7f1c6c035a80 msgr2=0x7f1c6c037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 1 --2- 172.21.15.104:0/16684548 >> v2:172.21.15.174:6812/17603 conn(0x7f1c6c035a80 0x7f1c6c037f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c006f00 tx=0x7f1c7c008040 comp rx=0 tx=0).stop 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 10 monclient: shutdown 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1c8c132120 msgr2=0x7f1c8c130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8b59e700 1 --2- 172.21.15.104:0/16684548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1c8c132120 0x7f1c8c130fc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400c4e0 tx=0x7f1c74007ea0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.993+0000 7f1c8ad9d700 1 -- 172.21.15.104:0/16684548 reap_dead start 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.997+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 shutdown_connections 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.997+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 >> 172.21.15.104:0/16684548 conn(0x7f1c8c068120 msgr2=0x7f1c8c0a0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.997+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 shutdown_connections 2024-09-18T14:27:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:56.997+0000 7f1c8b59e700 1 -- 172.21.15.104:0/16684548 wait complete. 2024-09-18T14:27:57.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-18T14:27:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient: get_monmap_and_config 2024-09-18T14:27:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient: monmap: 2024-09-18T14:27:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:57.121791+0000 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:57.121791+0000 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:57.124 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 1 Processor -- start 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 1 -- start start 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient: init 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 0x7f7bb0097e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient(hunting): picked mon.noname-b con 0x7f7bb0097a50 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7bb009aac0 0x7f7bb009ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient(hunting): picked mon.noname-c con 0x7f7bb009aac0 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7bb009b3d0 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb009d7f0 con 0x7f7bb009b3d0 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb009da30 con 0x7f7bb009aac0 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0096a60 con 0x7f7bb0097a50 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.117+0000 7f7bb6ad3700 10 monclient(hunting): authenticate will time out at 1198.111826s 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 0x7f7bb0097e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7bb009aac0 0x7f7bb009ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 0x7f7bb0097e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43112/0 (socket says 172.21.15.104:43112) 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 -- 172.21.15.104:0/638399093 learned_addr learned my addr 172.21.15.104:0/638399093 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): get_auth_request con 0x7f7bb0097a50 auth_method 0 2024-09-18T14:27:57.128 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb5070700 1 --1- 172.21.15.104:0/638399093 >> v1:172.21.15.174:6789/0 conn(0x7f7bb009b3d0 0x7f7bb009bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.104:47438/0 (socket says 172.21.15.104:47438) 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 10 monclient(hunting): get_auth_request con 0x7f7bb009aac0 auth_method 0 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.129 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient(hunting): handle_auth_done global_id 34190 payload 274 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 -- 172.21.15.104:0/638399093 >> v2:172.21.15.174:3300/0 conn(0x7f7bb009aac0 msgr2=0x7f7bb009ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 --2- 172.21.15.104:0/638399093 >> v2:172.21.15.174:3300/0 conn(0x7f7bb009aac0 0x7f7bb009ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 -- 172.21.15.104:0/638399093 >> v1:172.21.15.174:6789/0 conn(0x7f7bb009b3d0 legacy=0x7f7bb009bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 monclient: found mon.noname-b 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:57.130 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 -- 172.21.15.104:0/638399093 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0098770 con 0x7f7bb0097a50 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 -- 172.21.15.104:0/638399093 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 msgr2=0x7f7bb0097e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb486f700 1 --2- 172.21.15.104:0/638399093 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 0x7f7bb0097e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b980099b0 tx=0x7f7b98009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d05c59a37ca421e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 1 -- 172.21.15.104:0/638399093 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98009ea0 con 0x7f7bb0097a50 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: dump: 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.133 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb6ad3700 5 monclient: authenticate success, global_id 34190 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb6ad3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 1 -- 172.21.15.104:0/638399093 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9800ba70 con 0x7f7bb0097a50 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb6ad3700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:57.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7bb6ad3700 4 set_mon_vals no callback set 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 1 -- 172.21.15.104:0/638399093 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9800be30 con 0x7f7bb0097a50 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.121+0000 7f7baf7fe700 10 monclient: dump: 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.135 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient: shutdown 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/638399093 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 msgr2=0x7f7bb0097e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/638399093 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 0x7f7bb0097e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b980099b0 tx=0x7f7b98009250 comp rx=0 tx=0).stop 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/638399093 shutdown_connections 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/638399093 >> v2:172.21.15.174:3300/0 conn(0x7f7bb009aac0 0x7f7bb009ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/638399093 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb0097a50 0x7f7bb0097e20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/638399093 >> 172.21.15.104:0/638399093 conn(0x7f7bb00a1290 msgr2=0x7f7bb00a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/638399093 shutdown_connections 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/638399093 wait complete. 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient: monmap: 2024-09-18T14:27:57.136 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 Processor -- start 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- start start 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient: init 2024-09-18T14:27:57.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb0097a50 0x7f7bb01457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): picked mon.noname-b con 0x7f7bb0097a50 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb009aac0 0x7f7bb0138470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): picked mon.noname-c con 0x7f7bb009aac0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 0x7f7bb0121f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): picked mon.noname-a con 0x7f7bb009b3d0 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0125870 con 0x7f7bb009b3d0 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb006f5a0 con 0x7f7bb0097a50 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb009e970 con 0x7f7bb009aac0 2024-09-18T14:27:57.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb6ad3700 10 monclient(hunting): authenticate will time out at 1198.118528s 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 0x7f7bb0121f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb009aac0 0x7f7bb0138470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 0x7f7bb0121f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35820/0 (socket says 172.21.15.104:35820) 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 -- 172.21.15.104:0/2241164275 learned_addr learned my addr 172.21.15.104:0/2241164275 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): get_auth_request con 0x7f7bb009b3d0 auth_method 0 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 10 monclient(hunting): get_auth_request con 0x7f7bb009aac0 auth_method 0 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb486f700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb0097a50 0x7f7bb01457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb486f700 10 monclient(hunting): get_auth_request con 0x7f7bb0097a50 auth_method 0 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb486f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb486f700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb486f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient(hunting): handle_auth_done global_id 15246 payload 824 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 -- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb009aac0 msgr2=0x7f7bb0138470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb009aac0 0x7f7bb0138470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 -- 172.21.15.104:0/2241164275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb0097a50 msgr2=0x7f7bb01457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb0097a50 0x7f7bb01457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 monclient: found mon.noname-a 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7bb5070700 1 -- 172.21.15.104:0/2241164275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980094f0 con 0x7f7bb009b3d0 2024-09-18T14:27:57.141 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.125+0000 7f7baffff700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb009aac0 0x7f7bb0138470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:57.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb5070700 1 -- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 msgr2=0x7f7bb0121f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb5070700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 0x7f7bb0121f80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000c560 tx=0x7f7ba0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 1 -- 172.21.15.104:0/2241164275 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0005c80 con 0x7f7bb009b3d0 2024-09-18T14:27:57.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:57.143 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: dump: 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 5 monclient: authenticate success, global_id 15246 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 1 -- 172.21.15.104:0/2241164275 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba0005e00 con 0x7f7bb009b3d0 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 10 monclient: _renew_subs 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 1 -- 172.21.15.104:0/2241164275 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0007320 con 0x7f7bb009b3d0 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb0125590 con 0x7f7bb009b3d0 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7baeffd700 4 set_mon_vals no callback set 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 10 monclient: dump: 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.145 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 10 monclient: _renew_subs 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb0145ce0 con 0x7f7bb009b3d0 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.146 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb0064b30 con 0x7f7bb009b3d0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.129+0000 7f7bacff9700 1 -- 172.21.15.104:0/2241164275 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f7ba001b3b0 con 0x7f7bb009b3d0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.133+0000 7f7bacff9700 1 --2- 172.21.15.104:0/2241164275 >> v2:172.21.15.174:6812/17603 conn(0x7f7b90035a80 0x7f7b90037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.133+0000 7f7bacff9700 1 -- 172.21.15.104:0/2241164275 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7ba0047450 con 0x7f7bb009b3d0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.133+0000 7f7bb486f700 1 --2- 172.21.15.104:0/2241164275 >> v2:172.21.15.174:6812/17603 conn(0x7f7b90035a80 0x7f7b90037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.133+0000 7f7bb486f700 10 monclient: get_auth_request con 0x7f7b90035a80 auth_method 0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.133+0000 7f7bb486f700 1 --2- 172.21.15.104:0/2241164275 >> v2:172.21.15.174:6812/17603 conn(0x7f7b90035a80 0x7f7b90037f30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7b980099b0 tx=0x7f7b98016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.137+0000 7f7bacff9700 1 -- 172.21.15.104:0/2241164275 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7ba00196e0 con 0x7f7bb009b3d0 2024-09-18T14:27:57.147 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.137+0000 7f7bacff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.148 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.137+0000 7f7bacff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:57.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bb0125ab0 con 0x7f7b90035a80 2024-09-18T14:27:57.315 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bacff9700 1 -- 172.21.15.104:0/2241164275 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7b9800bbe0 con 0x7f7b90035a80 2024-09-18T14:27:57.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 >> v2:172.21.15.174:6812/17603 conn(0x7f7b90035a80 msgr2=0x7f7b90037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/2241164275 >> v2:172.21.15.174:6812/17603 conn(0x7f7b90035a80 0x7f7b90037f30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7b980099b0 tx=0x7f7b98016000 comp rx=0 tx=0).stop 2024-09-18T14:27:57.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bb6ad3700 10 monclient: shutdown 2024-09-18T14:27:57.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bb6ad3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 msgr2=0x7f7bb0121f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.309+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 0x7f7bb0121f80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000c560 tx=0x7f7ba0007f90 comp rx=0 tx=0).stop 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 shutdown_connections 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/2241164275 >> v2:172.21.15.174:6812/17603 conn(0x7f7b90035a80 0x7f7b90037f30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bb009b3d0 0x7f7bb0121f80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7bb009aac0 0x7f7bb0138470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 --2- 172.21.15.104:0/2241164275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb0097a50 0x7f7bb01457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 >> 172.21.15.104:0/2241164275 conn(0x7f7bb00a1290 msgr2=0x7f7bb0139370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 shutdown_connections 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.313+0000 7f7bb6ad3700 1 -- 172.21.15.104:0/2241164275 wait complete. 2024-09-18T14:27:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 4 to deep-scrub 2024-09-18T14:27:57.328 INFO:tasks.ceph:Scrubbing osd.5 2024-09-18T14:27:57.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:27:57.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient: get_monmap_and_config 2024-09-18T14:27:57.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient: monmap: 2024-09-18T14:27:57.439 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.439 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.439 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:57.437313+0000 2024-09-18T14:27:57.439 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:57.437313+0000 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:57.440 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.444 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 1 Processor -- start 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 1 -- start start 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient: init 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f98980a2a10 0x7f9898132230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient(hunting): picked mon.noname-c con 0x7f98980a2a10 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 0x7f989809f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient(hunting): picked mon.noname-b con 0x7f989809f300 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98980a1210 addr v1:172.21.15.174:6789/0 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989809d020 con 0x7f98980a1210 2024-09-18T14:27:57.445 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98980a2de0 con 0x7f98980a2a10 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989809fc40 con 0x7f989809f300 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f98a010f700 10 monclient(hunting): authenticate will time out at 1198.427506s 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989d6aa700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 0x7f989809f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989d6aa700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 0x7f989809f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43132/0 (socket says 172.21.15.104:43132) 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989d6aa700 1 -- 172.21.15.104:0/741447046 learned_addr learned my addr 172.21.15.104:0/741447046 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989d6aa700 10 monclient(hunting): get_auth_request con 0x7f989809f300 auth_method 0 2024-09-18T14:27:57.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989d6aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989d6aa700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989d6aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.433+0000 7f989deab700 1 --2- 172.21.15.104:0/741447046 >> v2:172.21.15.174:3300/0 conn(0x7f98980a2a10 0x7f9898132230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989deab700 10 monclient(hunting): get_auth_request con 0x7f98980a2a10 auth_method 0 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989deab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989deab700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989deab700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989deab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989deab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989deab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 10 monclient(hunting): handle_auth_done global_id 34196 payload 274 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 1 -- 172.21.15.104:0/741447046 >> v2:172.21.15.174:3300/0 conn(0x7f98980a2a10 msgr2=0x7f9898132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 1 --2- 172.21.15.104:0/741447046 >> v2:172.21.15.174:3300/0 conn(0x7f98980a2a10 0x7f9898132230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 1 -- 172.21.15.104:0/741447046 >> v1:172.21.15.174:6789/0 conn(0x7f98980a1210 legacy=0x7f98980a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 1 monclient: found mon.noname-b 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:57.448 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 1 -- 172.21.15.104:0/741447046 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989809d220 con 0x7f989809f300 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 1 -- 172.21.15.104:0/741447046 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 msgr2=0x7f989809f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989d6aa700 1 --2- 172.21.15.104:0/741447046 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 0x7f989809f700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f988c0099b0 tx=0x7f988c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4f56e832698f6e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 1 -- 172.21.15.104:0/741447046 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c009ea0 con 0x7f989809f300 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: dump: 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.449 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 1 -- 172.21.15.104:0/741447046 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988c00ba70 con 0x7f989809f300 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f98a010f700 5 monclient: authenticate success, global_id 34196 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f98a010f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 1 -- 172.21.15.104:0/741447046 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c00be30 con 0x7f989809f300 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f98a010f700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f98a010f700 4 set_mon_vals no callback set 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.437+0000 7f989cea9700 10 monclient: dump: 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.451 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.452 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.452 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.452 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.452 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.452 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient: shutdown 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- 172.21.15.104:0/741447046 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 msgr2=0x7f989809f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 --2- 172.21.15.104:0/741447046 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 0x7f989809f700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f988c0099b0 tx=0x7f988c009250 comp rx=0 tx=0).stop 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- 172.21.15.104:0/741447046 shutdown_connections 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 --2- 172.21.15.104:0/741447046 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f989809f300 0x7f989809f700 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 --2- 172.21.15.104:0/741447046 >> v2:172.21.15.174:3300/0 conn(0x7f98980a2a10 0x7f9898132230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- 172.21.15.104:0/741447046 >> 172.21.15.104:0/741447046 conn(0x7f98980a6290 msgr2=0x7f98980a2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- 172.21.15.104:0/741447046 shutdown_connections 2024-09-18T14:27:57.458 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- 172.21.15.104:0/741447046 wait complete. 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient: monmap: 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:57.459 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 Processor -- start 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- start start 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient: init 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f989809f300 0x7f989809b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): picked mon.noname-b con 0x7f989809f300 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98980a1210 0x7f989809bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.460 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): picked mon.noname-a con 0x7f98980a1210 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f98980a2a10 0x7f989812ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): picked mon.noname-c con 0x7f98980a2a10 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989812f6d0 con 0x7f98980a1210 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989809dfb0 con 0x7f989809f300 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9898134ee0 con 0x7f98980a2a10 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f98a010f700 10 monclient(hunting): authenticate will time out at 1198.434049s 2024-09-18T14:27:57.461 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98980a1210 0x7f989809bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f98980a2a10 0x7f989812ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98980a1210 0x7f989809bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35836/0 (socket says 172.21.15.104:35836) 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 -- 172.21.15.104:0/3852904282 learned_addr learned my addr 172.21.15.104:0/3852904282 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): get_auth_request con 0x7f98980a1210 auth_method 0 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 10 monclient(hunting): get_auth_request con 0x7f98980a2a10 auth_method 0 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 1 --2- 172.21.15.104:0/3852904282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f989809f300 0x7f989809b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 10 monclient(hunting): get_auth_request con 0x7f989809f300 auth_method 0 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989e6ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989deab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient(hunting): handle_auth_done global_id 15249 payload 824 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 -- 172.21.15.104:0/3852904282 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f98980a2a10 msgr2=0x7f989812ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 --2- 172.21.15.104:0/3852904282 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f98980a2a10 0x7f989812ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 -- 172.21.15.104:0/3852904282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f989809f300 msgr2=0x7f989809b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 --2- 172.21.15.104:0/3852904282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f989809f300 0x7f989809b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 monclient: found mon.noname-a 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.441+0000 7f989d6aa700 1 -- 172.21.15.104:0/3852904282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0094f0 con 0x7f98980a1210 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f989d6aa700 1 -- 172.21.15.104:0/3852904282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98980a1210 msgr2=0x7f989809bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f989d6aa700 1 --2- 172.21.15.104:0/3852904282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98980a1210 0x7f989809bd80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f988c006fd0 tx=0x7f988c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c006b60 con 0x7f98980a1210 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: dump: 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.465 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 5 monclient: authenticate success, global_id 15249 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988c020070 con 0x7f98980a1210 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 10 monclient: _renew_subs 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989812db30 con 0x7f98980a1210 2024-09-18T14:27:57.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c011d50 con 0x7f98980a1210 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 10 monclient: dump: 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.467 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 10 monclient: _renew_subs 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98977fe700 4 set_mon_vals no callback set 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9898130330 con 0x7f98980a1210 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f989812d670 con 0x7f98980a1210 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.445+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f988c022030 con 0x7f98980a1210 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f9895ffb700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6812/17603 conn(0x7f9884035a80 0x7f9884037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f988c0498f0 con 0x7f98980a1210 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f9895ffb700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6804/17846 conn(0x7f988403b5a0 0x7f988403d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 --> v2:172.21.15.174:6804/17846 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f988403e020 con 0x7f988403b5a0 2024-09-18T14:27:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=327) v2 ==== 24+0+0 (secure 0 0 0) 0x7f988c04e2a0 con 0x7f98980a1210 2024-09-18T14:27:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f9895ffb700 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-18T14:27:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f989deab700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6812/17603 conn(0x7f9884035a80 0x7f9884037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f989deab700 10 monclient: get_auth_request con 0x7f9884035a80 auth_method 0 2024-09-18T14:27:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f989e6ac700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6804/17846 conn(0x7f988403b5a0 0x7f988403d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f989e6ac700 10 monclient: get_auth_request con 0x7f988403b5a0 auth_method 0 2024-09-18T14:27:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.449+0000 7f989deab700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6812/17603 conn(0x7f9884035a80 0x7f9884037f30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9888009fd0 tx=0x7f988800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.453+0000 7f989e6ac700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6804/17846 conn(0x7f988403b5a0 0x7f988403d9b0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.470 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.453+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== osd.5 v2:172.21.15.174:6804/17846 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f988403e020 con 0x7f988403b5a0 2024-09-18T14:27:57.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 --> v2:172.21.15.174:6804/17846 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f989812dad0 con 0x7f988403b5a0 2024-09-18T14:27:57.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f9895ffb700 1 -- 172.21.15.104:0/3852904282 <== osd.5 v2:172.21.15.174:6804/17846 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f989812dad0 con 0x7f988403b5a0 2024-09-18T14:27:57.480 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-18T14:27:57.480 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:27:57.480 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6804/17846 conn(0x7f988403b5a0 msgr2=0x7f988403d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6804/17846 conn(0x7f988403b5a0 0x7f988403d9b0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6812/17603 conn(0x7f9884035a80 msgr2=0x7f9884037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 1 --2- 172.21.15.104:0/3852904282 >> v2:172.21.15.174:6812/17603 conn(0x7f9884035a80 0x7f9884037f30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9888009fd0 tx=0x7f988800b040 comp rx=0 tx=0).stop 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 10 monclient: shutdown 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98980a1210 msgr2=0x7f989809bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f98a010f700 1 --2- 172.21.15.104:0/3852904282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98980a1210 0x7f989809bd80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f988c006fd0 tx=0x7f988c016000 comp rx=0 tx=0).stop 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.473+0000 7f989e6ac700 1 -- 172.21.15.104:0/3852904282 reap_dead start 2024-09-18T14:27:57.482 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.477+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 shutdown_connections 2024-09-18T14:27:57.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.477+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 >> 172.21.15.104:0/3852904282 conn(0x7f98980a6290 msgr2=0x7f98980a41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.477+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 shutdown_connections 2024-09-18T14:27:57.483 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.477+0000 7f98a010f700 1 -- 172.21.15.104:0/3852904282 wait complete. 2024-09-18T14:27:57.493 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-18T14:27:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.597+0000 7f7b71d38700 10 monclient: get_monmap_and_config 2024-09-18T14:27:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.597+0000 7f7b71d38700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.597+0000 7f7b71d38700 10 monclient: monmap: 2024-09-18T14:27:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:57.603228+0000 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:57.603228+0000 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 1 Processor -- start 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 1 -- start start 2024-09-18T14:27:57.606 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient: init 2024-09-18T14:27:57.610 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 0x7f7b6c0999b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c09c470 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7b6c09afb0 0x7f7b6c09c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c09afb0 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b6c09cd80 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c06a540 con 0x7f7b6c09cd80 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6c09b380 con 0x7f7b6c09afb0 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c09a5d0 con 0x7f7b6c09c470 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71d38700 10 monclient(hunting): authenticate will time out at 1198.593128s 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 0x7f7b6c0999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b63fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7b6c09afb0 0x7f7b6c09c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 0x7f7b6c0999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43146/0 (socket says 172.21.15.104:43146) 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 -- 172.21.15.104:0/2681640392 learned_addr learned my addr 172.21.15.104:0/2681640392 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b71537700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7b6c09cd80 0x7f7b6c0940b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34840/0 (socket says 172.21.15.104:34840) 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): get_auth_request con 0x7f7b6c09c470 auth_method 0 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.612 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b63fff700 10 monclient(hunting): get_auth_request con 0x7f7b6c09afb0 auth_method 0 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b63fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient(hunting): handle_auth_done global_id 34202 payload 274 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 -- 172.21.15.104:0/2681640392 >> v2:172.21.15.104:3300/0 conn(0x7f7b6c09afb0 msgr2=0x7f7b6c09c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 --2- 172.21.15.104:0/2681640392 >> v2:172.21.15.104:3300/0 conn(0x7f7b6c09afb0 0x7f7b6c09c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 -- 172.21.15.104:0/2681640392 >> v1:172.21.15.104:6789/0 conn(0x7f7b6c09cd80 legacy=0x7f7b6c0940b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 monclient: found mon.noname-b 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.601+0000 7f7b70d36700 1 -- 172.21.15.104:0/2681640392 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c09a8b0 con 0x7f7b6c09c470 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b70d36700 1 -- 172.21.15.104:0/2681640392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 msgr2=0x7f7b6c0999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b70d36700 1 --2- 172.21.15.104:0/2681640392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 0x7f7b6c0999b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800bc80 tx=0x7f7b6800b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dec24205cc0696c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 1 -- 172.21.15.104:0/2681640392 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6800be90 con 0x7f7b6c09c470 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: dump: 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.614 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 1 -- 172.21.15.104:0/2681640392 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b68010470 con 0x7f7b6c09c470 2024-09-18T14:27:57.615 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 5 monclient: authenticate success, global_id 34202 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 1 -- 172.21.15.104:0/2681640392 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68010830 con 0x7f7b6c09c470 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 4 set_mon_vals no callback set 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b637fe700 10 monclient: dump: 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.616 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 10 monclient: shutdown 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 -- 172.21.15.104:0/2681640392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 msgr2=0x7f7b6c0999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 --2- 172.21.15.104:0/2681640392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 0x7f7b6c0999b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800bc80 tx=0x7f7b6800b4f0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.617 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 -- 172.21.15.104:0/2681640392 shutdown_connections 2024-09-18T14:27:57.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 --2- 172.21.15.104:0/2681640392 >> v2:172.21.15.104:3300/0 conn(0x7f7b6c09afb0 0x7f7b6c09c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 --2- 172.21.15.104:0/2681640392 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09c470 0x7f7b6c0999b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 -- 172.21.15.104:0/2681640392 >> 172.21.15.104:0/2681640392 conn(0x7f7b6c0a11f0 msgr2=0x7f7b6c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 -- 172.21.15.104:0/2681640392 shutdown_connections 2024-09-18T14:27:57.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 -- 172.21.15.104:0/2681640392 wait complete. 2024-09-18T14:27:57.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.624 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 10 monclient: monmap: 2024-09-18T14:27:57.624 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.624 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:57.624 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:57.624 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 Processor -- start 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 1 -- start start 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.605+0000 7f7b71d38700 10 monclient: init 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:27:57.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 0x7f7b6c08cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c09afb0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b6c09c470 0x7f7b6c08d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c09c470 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b6c09cd80 0x7f7b6c08d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c09cd80 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c090380 con 0x7f7b6c09c470 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c091330 con 0x7f7b6c09cd80 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.626 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c090660 con 0x7f7b6c09afb0 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71d38700 10 monclient(hunting): authenticate will time out at 1198.600789s 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 0x7f7b6c08cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b6c09c470 0x7f7b6c08d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 0x7f7b6c08cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:43160/0 (socket says 172.21.15.104:43160) 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b6c09c470 0x7f7b6c08d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35868/0 (socket says 172.21.15.104:35868) 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 -- 172.21.15.104:0/1113012706 learned_addr learned my addr 172.21.15.104:0/1113012706 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): get_auth_request con 0x7f7b6c09afb0 auth_method 0 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 10 monclient(hunting): get_auth_request con 0x7f7b6c09c470 auth_method 0 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71537700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b6c09cd80 0x7f7b6c08d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71537700 10 monclient(hunting): get_auth_request con 0x7f7b6c09cd80 auth_method 0 2024-09-18T14:27:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71537700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71537700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71537700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient(hunting): handle_auth_done global_id 34205 payload 824 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 -- 172.21.15.104:0/1113012706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b6c09cd80 msgr2=0x7f7b6c08d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b6c09cd80 0x7f7b6c08d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 -- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b6c09c470 msgr2=0x7f7b6c08d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b6c09c470 0x7f7b6c08d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 monclient: found mon.noname-c 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.104:3301/0 2024-09-18T14:27:57.629 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 -- 172.21.15.104:0/1113012706 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6800b790 con 0x7f7b6c09afb0 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b71537700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b6c09cd80 0x7f7b6c08d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 -- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 msgr2=0x7f7b6c08cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.609+0000 7f7b70d36700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 0x7f7b6c08cc80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c09a710 tx=0x7f7b6801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 1 -- 172.21.15.104:0/1113012706 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68018630 con 0x7f7b6c09afb0 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: dump: 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.630 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 5 monclient: authenticate success, global_id 34205 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 1 -- 172.21.15.104:0/1113012706 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b68018790 con 0x7f7b6c09afb0 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 1 -- 172.21.15.104:0/1113012706 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68018b10 con 0x7f7b6c09afb0 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 10 monclient: _renew_subs 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c08fd70 con 0x7f7b6c09afb0 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 10 monclient: dump: 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.632 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b62ffd700 4 set_mon_vals no callback set 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 10 monclient: _renew_subs 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c0900a0 con 0x7f7b6c09afb0 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 1 -- 172.21.15.104:0/1113012706 <== mon.2 v2:172.21.15.104:3301/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f7b68023420 con 0x7f7b6c09afb0 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.104:3301/0 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c099ee0 con 0x7f7b6c09afb0 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 1 --2- 172.21.15.104:0/1113012706 >> v2:172.21.15.174:6812/17603 conn(0x7f7b58035a80 0x7f7b58037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b60ff9700 1 -- 172.21.15.104:0/1113012706 <== mon.2 v2:172.21.15.104:3301/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7b68014070 con 0x7f7b6c09afb0 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b63fff700 1 --2- 172.21.15.104:0/1113012706 >> v2:172.21.15.174:6812/17603 conn(0x7f7b58035a80 0x7f7b58037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.613+0000 7f7b63fff700 10 monclient: get_auth_request con 0x7f7b58035a80 auth_method 0 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.617+0000 7f7b63fff700 1 --2- 172.21.15.104:0/1113012706 >> v2:172.21.15.174:6812/17603 conn(0x7f7b58035a80 0x7f7b58037f30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c006fd0 tx=0x7f7b5c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.634 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.625+0000 7f7b60ff9700 1 -- 172.21.15.104:0/1113012706 <== mon.2 v2:172.21.15.104:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7b6807c670 con 0x7f7b6c09afb0 2024-09-18T14:27:57.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.625+0000 7f7b60ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.635 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.625+0000 7f7b60ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:57.786 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.781+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b6c06e650 con 0x7f7b58035a80 2024-09-18T14:27:57.787 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.781+0000 7f7b60ff9700 1 -- 172.21.15.104:0/1113012706 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7b5c0107e0 con 0x7f7b58035a80 2024-09-18T14:27:57.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 >> v2:172.21.15.174:6812/17603 conn(0x7f7b58035a80 msgr2=0x7f7b58037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 --2- 172.21.15.104:0/1113012706 >> v2:172.21.15.174:6812/17603 conn(0x7f7b58035a80 0x7f7b58037f30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c006fd0 tx=0x7f7b5c008040 comp rx=0 tx=0).stop 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 10 monclient: shutdown 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 msgr2=0x7f7b6c08cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 0x7f7b6c08cc80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c09a710 tx=0x7f7b6801a000 comp rx=0 tx=0).stop 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 shutdown_connections 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 --2- 172.21.15.104:0/1113012706 >> v2:172.21.15.174:6812/17603 conn(0x7f7b58035a80 0x7f7b58037f30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b6c09cd80 0x7f7b6c08d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b6c09c470 0x7f7b6c08d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 --2- 172.21.15.104:0/1113012706 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f7b6c09afb0 0x7f7b6c08cc80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 >> 172.21.15.104:0/1113012706 conn(0x7f7b6c0a11f0 msgr2=0x7f7b6c06ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 shutdown_connections 2024-09-18T14:27:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.785+0000 7f7b71d38700 1 -- 172.21.15.104:0/1113012706 wait complete. 2024-09-18T14:27:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 5 to deep-scrub 2024-09-18T14:27:57.802 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:57.901 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient: get_monmap_and_config 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient: monmap: 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:27:57.899963+0000 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:27:57.899963+0000 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:27:57.902 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:27:57.903 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 1 Processor -- start 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 1 -- start start 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient: init 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f411c0a27a0 addr v1:172.21.15.104:6789/0 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f411c09bc90 0x7f411c09c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient(hunting): picked mon.noname-c con 0x7f411c09bc90 addr v2:172.21.15.174:3300/0 2024-09-18T14:27:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f411c136c80 0x7f411c139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient(hunting): picked mon.noname-a con 0x7f411c136c80 addr v2:172.21.15.104:3300/0 2024-09-18T14:27:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c099610 con 0x7f411c0a27a0 2024-09-18T14:27:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411c099880 con 0x7f411c136c80 2024-09-18T14:27:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411c0a2b70 con 0x7f411c09bc90 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122e64700 10 monclient(hunting): authenticate will time out at 1198.889679s 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121e62700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f411c0a27a0 0x7f411c09a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34848/0 (socket says 172.21.15.104:34848) 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121e62700 1 -- 172.21.15.104:0/2197452166 learned_addr learned my addr 172.21.15.104:0/2197452166 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122663700 1 --2- 172.21.15.104:0/2197452166 >> v2:172.21.15.104:3300/0 conn(0x7f411c136c80 0x7f411c139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122663700 10 monclient(hunting): get_auth_request con 0x7f411c136c80 auth_method 0 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122663700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122663700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4122663700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 1 --2- 172.21.15.104:0/2197452166 >> v2:172.21.15.174:3300/0 conn(0x7f411c09bc90 0x7f411c09c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): get_auth_request con 0x7f411c09bc90 auth_method 0 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient(hunting): handle_auth_done global_id 5905 payload 274 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 1 -- 172.21.15.104:0/2197452166 >> v2:172.21.15.104:3300/0 conn(0x7f411c136c80 msgr2=0x7f411c139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 1 --2- 172.21.15.104:0/2197452166 >> v2:172.21.15.104:3300/0 conn(0x7f411c136c80 0x7f411c139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 1 -- 172.21.15.104:0/2197452166 >> v1:172.21.15.104:6789/0 conn(0x7f411c0a27a0 legacy=0x7f411c09a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 1 monclient: found mon.noname-c 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.897+0000 7f4121661700 1 -- 172.21.15.104:0/2197452166 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c09e960 con 0x7f411c09bc90 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4121661700 1 -- 172.21.15.104:0/2197452166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c09bc90 msgr2=0x7f411c09c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4121661700 1 --2- 172.21.15.104:0/2197452166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c09bc90 0x7f411c09c090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f411c09dc60 tx=0x7f41180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d692aebf15dde19 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 1 -- 172.21.15.104:0/2197452166 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800b8c0 con 0x7f411c09bc90 2024-09-18T14:27:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: dump: 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.912 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 1 -- 172.21.15.104:0/2197452166 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800ba20 con 0x7f411c09bc90 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 1 -- 172.21.15.104:0/2197452166 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800bdc0 con 0x7f411c09bc90 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4120e60700 10 monclient: dump: 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.914 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 5 monclient: authenticate success, global_id 5905 2024-09-18T14:27:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 10 monclient: get_monmap_and_config success 2024-09-18T14:27:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 4 set_mon_vals no callback set 2024-09-18T14:27:57.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 10 monclient: shutdown 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 -- 172.21.15.104:0/2197452166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c09bc90 msgr2=0x7f411c09c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 --2- 172.21.15.104:0/2197452166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c09bc90 0x7f411c09c090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f411c09dc60 tx=0x7f41180091a0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 -- 172.21.15.104:0/2197452166 shutdown_connections 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 --2- 172.21.15.104:0/2197452166 >> v2:172.21.15.104:3300/0 conn(0x7f411c136c80 0x7f411c139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 --2- 172.21.15.104:0/2197452166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c09bc90 0x7f411c09c090 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 -- 172.21.15.104:0/2197452166 >> 172.21.15.104:0/2197452166 conn(0x7f411c068e60 msgr2=0x7f411c0a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 -- 172.21.15.104:0/2197452166 shutdown_connections 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.901+0000 7f4122e64700 1 -- 172.21.15.104:0/2197452166 wait complete. 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient: build_initial_monmap 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient: monmap: 2024-09-18T14:27:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 Processor -- start 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 -- start start 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient: init 2024-09-18T14:27:57.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient: _reopen_session rank -1 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 0x7f411c06e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): picked mon.noname-a con 0x7f411c09bc90 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f411c0a27a0 0x7f411c11cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): picked mon.noname-c con 0x7f411c0a27a0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c136c80 0x7f411c11d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): picked mon.noname-b con 0x7f411c136c80 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c06efe0 con 0x7f411c09bc90 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c09f8e0 con 0x7f411c136c80 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): start opening mon connection 2024-09-18T14:27:57.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c11fc30 con 0x7f411c0a27a0 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): _renew_subs 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122e64700 10 monclient(hunting): authenticate will time out at 1198.898427s 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 0x7f411c06e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f411c0a27a0 0x7f411c11cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 0x7f411c06e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35890/0 (socket says 172.21.15.104:35890) 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 -- 172.21.15.104:0/3359789297 learned_addr learned my addr 172.21.15.104:0/3359789297 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): get_auth_request con 0x7f411c09bc90 auth_method 0 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 10 monclient(hunting): get_auth_request con 0x7f411c0a27a0 auth_method 0 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c136c80 0x7f411c11d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 10 monclient(hunting): get_auth_request con 0x7f411c136c80 auth_method 0 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4122663700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient(hunting): handle_auth_done global_id 15261 payload 824 2024-09-18T14:27:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 -- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f411c0a27a0 msgr2=0x7f411c11cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f411c0a27a0 0x7f411c11cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 -- 172.21.15.104:0/3359789297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c136c80 msgr2=0x7f411c11d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c136c80 0x7f411c11d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient: _finish_hunting 0 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 monclient: found mon.noname-a 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121e62700 1 -- 172.21.15.104:0/3359789297 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4118009420 con 0x7f411c09bc90 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.905+0000 7f4121661700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f411c0a27a0 0x7f411c11cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4121e62700 1 -- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 msgr2=0x7f411c06e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4121e62700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 0x7f411c06e850 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f410c00c5e0 tx=0x7f410c004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 1 -- 172.21.15.104:0/3359789297 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c005d40 con 0x7f411c09bc90 2024-09-18T14:27:57.928 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: dump: 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: _finish_auth 0 2024-09-18T14:27:57.929 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 1 -- 172.21.15.104:0/3359789297 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c0070d0 con 0x7f411c09bc90 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 5 monclient: authenticate success, global_id 15261 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 10 monclient: _renew_subs 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c11eb00 con 0x7f411c09bc90 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 1 -- 172.21.15.104:0/3359789297 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c007490 con 0x7f411c09bc90 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 10 monclient: dump: 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:27:57.930 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4113fff700 4 set_mon_vals no callback set 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 10 monclient: _renew_subs 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411c1369e0 con 0x7f411c09bc90 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 1 -- 172.21.15.104:0/3359789297 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f410c01b3b0 con 0x7f411c09bc90 2024-09-18T14:27:57.931 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 1 --2- 172.21.15.104:0/3359789297 >> v2:172.21.15.174:6812/17603 conn(0x7f4100035a80 0x7f4100037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4111ffb700 1 -- 172.21.15.104:0/3359789297 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f410c047340 con 0x7f411c09bc90 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411c11fa60 con 0x7f411c09bc90 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4121661700 1 --2- 172.21.15.104:0/3359789297 >> v2:172.21.15.174:6812/17603 conn(0x7f4100035a80 0x7f4100037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.909+0000 7f4121661700 10 monclient: get_auth_request con 0x7f4100035a80 auth_method 0 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.913+0000 7f4121661700 1 --2- 172.21.15.104:0/3359789297 >> v2:172.21.15.174:6812/17603 conn(0x7f4100035a80 0x7f4100037f30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f41180030c0 tx=0x7f4118016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.917+0000 7f4111ffb700 1 -- 172.21.15.104:0/3359789297 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f410c017070 con 0x7f411c09bc90 2024-09-18T14:27:57.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.917+0000 7f4111ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:27:57.933 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:57.917+0000 7f4111ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:27:58.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.081+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f411c0633b0 con 0x7f4100035a80 2024-09-18T14:27:58.090 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4111ffb700 1 -- 172.21.15.104:0/3359789297 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51743 (secure 0 0 0) 0x7f411c0633b0 con 0x7f4100035a80 2024-09-18T14:27:58.091 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:27:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 >> v2:172.21.15.174:6812/17603 conn(0x7f4100035a80 msgr2=0x7f4100037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 --2- 172.21.15.104:0/3359789297 >> v2:172.21.15.174:6812/17603 conn(0x7f4100035a80 0x7f4100037f30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f41180030c0 tx=0x7f4118016000 comp rx=0 tx=0).stop 2024-09-18T14:27:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 10 monclient: shutdown 2024-09-18T14:27:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:27:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 msgr2=0x7f411c06e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:27:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 0x7f411c06e850 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f410c00c5e0 tx=0x7f410c004a80 comp rx=0 tx=0).stop 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 shutdown_connections 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 --2- 172.21.15.104:0/3359789297 >> v2:172.21.15.174:6812/17603 conn(0x7f4100035a80 0x7f4100037f30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f411c136c80 0x7f411c11d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f411c0a27a0 0x7f411c11cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 --2- 172.21.15.104:0/3359789297 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f411c09bc90 0x7f411c06e850 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 >> 172.21.15.104:0/3359789297 conn(0x7f411c068e60 msgr2=0x7f411c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 shutdown_connections 2024-09-18T14:27:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:27:58.085+0000 7f4122e64700 1 -- 172.21.15.104:0/3359789297 wait complete. 2024-09-18T14:27:58.094 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:27:58.104 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-18T14:27:56.517409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4795856,"kb_used_data":557400424,"kb_used_omap":58312,"kb_used_meta":0,"kb_avail":557400424,"statfs":{"total":575688990720,"available":570778034176,"internally_reserved":0,"allocated":570778034176,"data_stored":570778034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59711673,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003056"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403327+0000","last_change":"2024-09-18T14:21:35.135892+0000","last_active":"2024-09-18T14:27:35.403327+0000","last_peered":"2024-09-18T14:27:35.403327+0000","last_clean":"2024-09-18T14:27:35.403327+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:27:35.403327+0000","last_undegraded":"2024-09-18T14:27:35.403327+0000","last_fullsized":"2024-09-18T14:27:35.403327+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:12:06.066321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.407340+0000","last_change":"2024-09-18T14:21:34.296145+0000","last_active":"2024-09-18T14:27:35.407340+0000","last_peered":"2024-09-18T14:27:35.407340+0000","last_clean":"2024-09-18T14:27:35.407340+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:27:35.407340+0000","last_undegraded":"2024-09-18T14:27:35.407340+0000","last_fullsized":"2024-09-18T14:27:35.407340+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:55.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.405428+0000","last_change":"2024-09-18T14:21:35.135757+0000","last_active":"2024-09-18T14:27:35.405428+0000","last_peered":"2024-09-18T14:27:35.405428+0000","last_clean":"2024-09-18T14:27:35.405428+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:27:35.405428+0000","last_undegraded":"2024-09-18T14:27:35.405428+0000","last_fullsized":"2024-09-18T14:27:35.405428+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:05:22.035355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.401849+0000","last_change":"2024-09-18T14:21:34.296326+0000","last_active":"2024-09-18T14:27:35.401849+0000","last_peered":"2024-09-18T14:27:35.401849+0000","last_clean":"2024-09-18T14:27:35.401849+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:27:35.401849+0000","last_undegraded":"2024-09-18T14:27:35.401849+0000","last_fullsized":"2024-09-18T14:27:35.401849+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:00:54.208144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":308,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404309+0000","last_change":"2024-09-18T14:21:34.296315+0000","last_active":"2024-09-18T14:27:35.404309+0000","last_peered":"2024-09-18T14:27:35.404309+0000","last_clean":"2024-09-18T14:27:35.404309+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:27:35.404309+0000","last_undegraded":"2024-09-18T14:27:35.404309+0000","last_fullsized":"2024-09-18T14:27:35.404309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:24:34.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":304,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.008862+0000","last_change":"2024-09-18T14:21:34.349809+0000","last_active":"2024-09-18T14:27:35.008862+0000","last_peered":"2024-09-18T14:27:35.008862+0000","last_clean":"2024-09-18T14:27:35.008862+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:27:35.008862+0000","last_undegraded":"2024-09-18T14:27:35.008862+0000","last_fullsized":"2024-09-18T14:27:35.008862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:15:09.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403712+0000","last_change":"2024-09-18T14:21:35.136264+0000","last_active":"2024-09-18T14:27:35.403712+0000","last_peered":"2024-09-18T14:27:35.403712+0000","last_clean":"2024-09-18T14:27:35.403712+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:27:35.403712+0000","last_undegraded":"2024-09-18T14:27:35.403712+0000","last_fullsized":"2024-09-18T14:27:35.403712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:42.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":307,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.404551+0000","last_change":"2024-09-18T14:21:34.296379+0000","last_active":"2024-09-18T14:27:35.404551+0000","last_peered":"2024-09-18T14:27:35.404551+0000","last_clean":"2024-09-18T14:27:35.404551+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:27:35.404551+0000","last_undegraded":"2024-09-18T14:27:35.404551+0000","last_fullsized":"2024-09-18T14:27:35.404551+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:31.260425+0000","last_clean_scrub_stamp":"2024-09-18T14:21:31.260425+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:13:56.421829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":403,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T14:27:35.403858+0000","last_change":"2024-09-18T14:21:28.174681+0000","last_active":"2024-09-18T14:27:35.403858+0000","last_peered":"2024-09-18T14:27:35.403858+0000","last_clean":"2024-09-18T14:27:35.403858+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:27:35.403858+0000","last_undegraded":"2024-09-18T14:27:35.403858+0000","last_fullsized":"2024-09-18T14:27:35.403858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:21:27.156511+0000","last_clean_scrub_stamp":"2024-09-18T14:21:27.156511+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:18:14.383506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9390,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9615786,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.77400000000000002,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.90200000000000002,"5min":1.204,"15min":1.204},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.55700000000000005,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.59799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47899999999999998},"max":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":1.002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91400000000000003,"15min":0.89600000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.38500000000000001},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":3.742},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.84299999999999997,"15min":0.85899999999999999},"min":{"1min":0.63400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.6240000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.98299999999999998,"15min":0.89500000000000002},"min":{"1min":0.93400000000000005,"5min":0.623,"15min":0.432},"max":{"1min":1.361,"5min":1.927,"15min":1.927},"last":0.95299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76400000000000001,"15min":0.79500000000000004},"min":{"1min":0.56799999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.2030000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.999,"15min":1.0309999999999999},"min":{"1min":0.74199999999999999,"5min":0.67700000000000005,"15min":0.57499999999999996},"max":{"1min":1.452,"5min":2.0590000000000002,"15min":4.0919999999999996},"last":0.88700000000000001}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76900000000000002,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.944,"5min":1.944,"15min":3.218},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.92600000000000005,"15min":1.006},"min":{"1min":0.72199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.5779999999999998},"last":0.84799999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799008,"kb_used_data":92900372,"kb_used_omap":9896,"kb_used_meta":0,"kb_avail":92900372,"statfs":{"total":95948165120,"available":95129980928,"internally_reserved":0,"allocated":95129980928,"data_stored":95129980928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10133777,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":1.3260000000000001,"5min":1.988,"15min":1.988},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.91000000000000003,"15min":0.97199999999999998},"min":{"1min":0.72199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.111,"5min":2.012,"15min":2.012},"last":0.876}]},{"osd":1,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.79300000000000004},"min":{"1min":0.66400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0980000000000001,"5min":2.714,"15min":2.714},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89100000000000001,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93999999999999995,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.875,"15min":0.91500000000000004},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.083,"5min":1.345,"15min":1.99},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94699999999999995,"15min":1.0109999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.046,"5min":1.361,"15min":2.1240000000000001},"last":0.83999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.69199999999999995,"15min":0.72799999999999998},"min":{"1min":0.56399999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.222},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1990000000000001,"15min":1.7769999999999999},"last":0.92000000000000004}]},{"osd":5,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73099999999999998,"15min":0.88700000000000001},"min":{"1min":0.56299999999999994,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.012,"5min":1.2070000000000001,"15min":3.0609999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84499999999999997,"15min":0.99099999999999999},"min":{"1min":0.65200000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.075,"5min":1.29,"15min":3.2799999999999998},"last":0.75900000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":9529,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9758155,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81399999999999995,"15min":0.93000000000000005},"min":{"1min":0.56100000000000005,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0109999999999999,"5min":3.8359999999999999,"15min":5.9100000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.98299999999999998},"min":{"1min":0.52500000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0429999999999999,"5min":3.5609999999999999,"15min":6.173},"last":0.72899999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.89600000000000002,"15min":1.034},"min":{"1min":0.53300000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.131,"5min":4.2519999999999998,"15min":5.8789999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.95599999999999996,"15min":0.89400000000000002},"min":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3140000000000001,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":0.80400000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90600000000000003,"15min":1.034},"min":{"1min":0.70399999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2250000000000001,"5min":5.9530000000000003,"15min":6.3109999999999999},"last":0.629},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.95999999999999996,"15min":1.0649999999999999},"min":{"1min":0.71899999999999997,"5min":0.499,"15min":0.499},"max":{"1min":1.0880000000000001,"5min":5.6159999999999997,"15min":6.2350000000000003},"last":0.69499999999999995}]},{"osd":4,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.095,"5min":2.2109999999999999,"15min":3.8420000000000001},"last":0.87},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79800000000000004,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.99399999999999999,"5min":2.4169999999999998,"15min":5.8410000000000002},"last":0.78100000000000003}]},{"osd":5,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.81599999999999995,"15min":0.93700000000000006},"min":{"1min":0.61199999999999999,"5min":0.53700000000000003,"15min":0.52100000000000002},"max":{"1min":1.167,"5min":1.581,"15min":4.0739999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92500000000000004,"15min":1.01},"min":{"1min":0.628,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.397,"5min":2.6899999999999999,"15min":4.0940000000000003},"last":0.91900000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799136,"kb_used_data":92900244,"kb_used_omap":9735,"kb_used_meta":0,"kb_avail":92900244,"statfs":{"total":95948165120,"available":95129849856,"internally_reserved":0,"allocated":95129849856,"data_stored":95129849856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9969199,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59399999999999997,"15min":0.56899999999999995},"min":{"1min":0.46700000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.878,"5min":1.849,"15min":1.849},"last":0.504},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.68600000000000005,"15min":0.59799999999999998},"min":{"1min":0.52900000000000003,"5min":0.377,"15min":0.376},"max":{"1min":1.375,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.53500000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78000000000000003,"15min":0.70199999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.206,"5min":4.7640000000000002,"15min":4.7640000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.84999999999999998,"15min":0.77700000000000002},"min":{"1min":0.56100000000000005,"5min":0.308,"15min":0.308},"max":{"1min":0.92800000000000005,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":0.41099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92900000000000005,"15min":0.89200000000000002},"min":{"1min":0.70499999999999996,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2350000000000001,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.92200000000000004},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.2729999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.67500000000000004}]},{"osd":4,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.91100000000000003,"15min":1.1830000000000001},"min":{"1min":0.69499999999999995,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.2949999999999999,"5min":1.778,"15min":9.0830000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95899999999999996,"15min":1.1359999999999999},"min":{"1min":0.75700000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.218,"5min":2.0089999999999999,"15min":7.1399999999999997},"last":0.74099999999999999}]},{"osd":5,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.96099999999999997,"15min":1.1850000000000001},"min":{"1min":0.79400000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.27,"5min":2.0859999999999999,"15min":9.4900000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.94699999999999995,"15min":1.155},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0369999999999999,"5min":2.2429999999999999,"15min":8.3420000000000005},"last":0.57799999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9693,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9926148,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.44700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":1.508,"15min":1.508},"last":0.435},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.28599999999999998},"max":{"1min":0.81599999999999995,"5min":1.038,"15min":1.038},"last":0.57299999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.251},"max":{"1min":0.92200000000000004,"5min":2.278,"15min":2.278},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74399999999999999,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.88900000000000001,"15min":1.022},"min":{"1min":0.66000000000000003,"5min":0.41499999999999998,"15min":0.34699999999999998},"max":{"1min":1.0109999999999999,"5min":2.8239999999999998,"15min":5.6699999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.95099999999999996,"15min":0.84899999999999998},"min":{"1min":0.67900000000000005,"5min":0.53500000000000003,"15min":0.32900000000000001},"max":{"1min":1.196,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.89000000000000001}]},{"osd":4,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88600000000000001,"15min":1.002},"min":{"1min":0.72199999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.079,"5min":1.45,"15min":3.7490000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.92900000000000005,"15min":0.86899999999999999},"min":{"1min":0.65300000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.018,"5min":6.141,"15min":6.141},"last":0.70799999999999996}]},{"osd":5,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.873,"15min":0.98999999999999999},"min":{"1min":0.70899999999999996,"5min":0.46800000000000003,"15min":0.40600000000000003},"max":{"1min":1.111,"5min":1.292,"15min":4.2229999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94199999999999995,"15min":0.91200000000000003},"min":{"1min":0.69899999999999995,"5min":0.46100000000000002,"15min":0.41799999999999998},"max":{"1min":1.198,"5min":1.496,"15min":1.496},"last":0.81200000000000006}]}]},{"osd":0,"up_from":11,"seq":47244640337,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":10067,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10308608,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31},"max":{"1min":0.753,"5min":1.298,"15min":1.298},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.80600000000000005,"5min":1.48,"15min":1.48},"last":0.371},{"interface":"front","average":{"1min":0.755,"5min":0.71199999999999997,"15min":0.69199999999999995},"min":{"1min":0.51400000000000001,"5min":0.46700000000000003,"15min":0.313},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.872,"15min":0.84499999999999997},"min":{"1min":0.65800000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1759999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.624},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.53100000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97499999999999998,"5min":1.212,"15min":1.212},"last":0.54500000000000004}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.97699999999999998},"min":{"1min":0.76900000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2470000000000001,"5min":1.579,"15min":1.9470000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.877,"15min":0.98299999999999998},"min":{"1min":0.627,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":2.476},"last":0.60699999999999998}]},{"osd":5,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.91300000000000003,"15min":0.89000000000000001},"min":{"1min":0.69299999999999995,"5min":0.56399999999999995,"15min":0.56299999999999994},"max":{"1min":1.2989999999999999,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93200000000000005,"15min":0.99399999999999999},"min":{"1min":0.73099999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1539999999999999,"5min":1.3009999999999999,"15min":1.7989999999999999},"last":0.70299999999999996}]}]}],"pool_statfs":[]}} 2024-09-18T14:27:58.109 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.109 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-18T14:21:31.260425+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T14:21:27.156511+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=21, tm_sec=27, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=27, tm_sec=53, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:27:58.110 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T14:28:18.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient: get_monmap_and_config 2024-09-18T14:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient: build_initial_monmap 2024-09-18T14:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient: monmap: 2024-09-18T14:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.226 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:28:18.224413+0000 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:28:18.224413+0000 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:28:18.227 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 1 Processor -- start 2024-09-18T14:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 1 -- start start 2024-09-18T14:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient: init 2024-09-18T14:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient: _reopen_session rank -1 2024-09-18T14:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T14:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f453809fe10 0x7f45380a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.230 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient(hunting): picked mon.noname-c con 0x7f453809fe10 addr v2:172.21.15.174:3300/0 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f453809c900 addr v1:172.21.15.174:6789/0 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 0x7f45380a0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient(hunting): picked mon.noname-b con 0x7f453809c090 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453809b990 con 0x7f453809c900 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45380a1320 con 0x7f453809fe10 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45380a0c60 con 0x7f453809c090 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient(hunting): _renew_subs 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453de00700 10 monclient(hunting): authenticate will time out at 1219.214531s 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 0x7f45380a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 0x7f45380a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3301/0 says I am v2:172.21.15.104:41378/0 (socket says 172.21.15.104:41378) 2024-09-18T14:28:18.231 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 -- 172.21.15.104:0/2055023267 learned_addr learned my addr 172.21.15.104:0/2055023267 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): get_auth_request con 0x7f453809c090 auth_method 0 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453cdfe700 1 --2- 172.21.15.104:0/2055023267 >> v2:172.21.15.174:3300/0 conn(0x7f453809fe10 0x7f45380a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453cdfe700 10 monclient(hunting): get_auth_request con 0x7f453809fe10 auth_method 0 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453cdfe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453cdfe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453cdfe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient(hunting): handle_auth_done global_id 34211 payload 274 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 -- 172.21.15.104:0/2055023267 >> v2:172.21.15.174:3300/0 conn(0x7f453809fe10 msgr2=0x7f45380a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 --2- 172.21.15.104:0/2055023267 >> v2:172.21.15.174:3300/0 conn(0x7f453809fe10 0x7f45380a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 -- 172.21.15.104:0/2055023267 >> v1:172.21.15.174:6789/0 conn(0x7f453809c900 legacy=0x7f4538131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient: _finish_hunting 0 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 monclient: found mon.noname-b 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.104:3301/0 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453d5ff700 1 -- 172.21.15.104:0/2055023267 --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453809b6b0 con 0x7f453809c090 2024-09-18T14:28:18.233 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.220+0000 7f453cdfe700 1 --2- 172.21.15.104:0/2055023267 >> v2:172.21.15.174:3300/0 conn(0x7f453809fe10 0x7f45380a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453d5ff700 1 -- 172.21.15.104:0/2055023267 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 msgr2=0x7f45380a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453d5ff700 1 --2- 172.21.15.104:0/2055023267 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 0x7f45380a0720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4530009f20 tx=0x7f4530009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd0ee857e7379444 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 1 -- 172.21.15.104:0/2055023267 <== mon.2 v2:172.21.15.104:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453000e070 con 0x7f453809c090 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: dump: 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.235 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: _finish_auth 0 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 5 monclient: authenticate success, global_id 34211 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T14:28:18.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 1 -- 172.21.15.104:0/2055023267 <== mon.2 v2:172.21.15.104:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453000bc90 con 0x7f453809c090 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 10 monclient: get_monmap_and_config success 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 4 set_mon_vals no callback set 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 1 -- 172.21.15.104:0/2055023267 <== mon.2 v2:172.21.15.104:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45300092e0 con 0x7f453809c090 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f45377fe700 10 monclient: dump: 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.237 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 10 monclient: shutdown 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 -- 172.21.15.104:0/2055023267 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 msgr2=0x7f45380a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 --2- 172.21.15.104:0/2055023267 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 0x7f45380a0720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4530009f20 tx=0x7f4530009840 comp rx=0 tx=0).stop 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 -- 172.21.15.104:0/2055023267 shutdown_connections 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 --2- 172.21.15.104:0/2055023267 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c090 0x7f45380a0720 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 --2- 172.21.15.104:0/2055023267 >> v2:172.21.15.174:3300/0 conn(0x7f453809fe10 0x7f45380a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 -- 172.21.15.104:0/2055023267 >> 172.21.15.104:0/2055023267 conn(0x7f45380a6080 msgr2=0x7f45380676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 -- 172.21.15.104:0/2055023267 shutdown_connections 2024-09-18T14:28:18.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.224+0000 7f453de00700 1 -- 172.21.15.104:0/2055023267 wait complete. 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient: build_initial_monmap 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient: monmap: 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 Processor -- start 2024-09-18T14:28:18.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 -- start start 2024-09-18T14:28:18.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient: init 2024-09-18T14:28:18.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient: _reopen_session rank -1 2024-09-18T14:28:18.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T14:28:18.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 0x7f453811e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): picked mon.noname-a con 0x7f453809c090 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c900 0x7f453811e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): picked mon.noname-c con 0x7f453809c900 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453809fe10 0x7f453811ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): picked mon.noname-b con 0x7f453809fe10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453809ae40 con 0x7f453809c090 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453809d060 con 0x7f453809fe10 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453811fd50 con 0x7f453809c900 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): _renew_subs 2024-09-18T14:28:18.245 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453de00700 10 monclient(hunting): authenticate will time out at 1219.222737s 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 0x7f453811e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 0x7f453811e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50502/0 (socket says 172.21.15.104:50502) 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 -- 172.21.15.104:0/1247080681 learned_addr learned my addr 172.21.15.104:0/1247080681 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): get_auth_request con 0x7f453809c090 auth_method 0 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c900 0x7f453811e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 10 monclient(hunting): get_auth_request con 0x7f453809c900 auth_method 0 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:28:18.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f4537fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453d5ff700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453809fe10 0x7f453811ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453d5ff700 10 monclient(hunting): get_auth_request con 0x7f453809fe10 auth_method 0 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453d5ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453d5ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453d5ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient(hunting): handle_auth_done global_id 15264 payload 824 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 -- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c900 msgr2=0x7f453811e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c900 0x7f453811e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 -- 172.21.15.104:0/1247080681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453809fe10 msgr2=0x7f453811ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453809fe10 0x7f453811ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient: _finish_hunting 0 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 monclient: found mon.noname-a 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.228+0000 7f453cdfe700 1 -- 172.21.15.104:0/1247080681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530009ae0 con 0x7f453809c090 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453cdfe700 1 -- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 msgr2=0x7f453811e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453cdfe700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 0x7f453811e480 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f452c00e8d0 tx=0x7f452c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 1 -- 172.21.15.104:0/1247080681 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c013620 con 0x7f453809c090 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: dump: 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.248 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: _finish_auth 0 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 1 -- 172.21.15.104:0/1247080681 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452c013780 con 0x7f453809c090 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453de00700 5 monclient: authenticate success, global_id 15264 2024-09-18T14:28:18.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453de00700 10 monclient: _renew_subs 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453de00700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4538121810 con 0x7f453809c090 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 1 -- 172.21.15.104:0/1247080681 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c013b20 con 0x7f453809c090 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 10 monclient: dump: 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453de00700 10 monclient: _renew_subs 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4536ffd700 4 set_mon_vals no callback set 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453de00700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4538121ba0 con 0x7f453809c090 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 1 -- 172.21.15.104:0/1247080681 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7f452c00c3a0 con 0x7f453809c090 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 1 --2- 172.21.15.104:0/1247080681 >> v2:172.21.15.174:6812/17603 conn(0x7f4524035a80 0x7f4524037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.232+0000 7f4534ff9700 1 -- 172.21.15.104:0/1247080681 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f452c047ca0 con 0x7f453809c090 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.236+0000 7f453de00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.236+0000 7f453de00700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.236+0000 7f453de00700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.236+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45380633b0 con 0x7f453809c090 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.236+0000 7f4537fff700 1 --2- 172.21.15.104:0/1247080681 >> v2:172.21.15.174:6812/17603 conn(0x7f4524035a80 0x7f4524037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.236+0000 7f4537fff700 10 monclient: get_auth_request con 0x7f4524035a80 auth_method 0 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.236+0000 7f4537fff700 1 --2- 172.21.15.104:0/1247080681 >> v2:172.21.15.174:6812/17603 conn(0x7f4524035a80 0x7f4524037f30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f453811f9e0 tx=0x7f4528007ea0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:28:18.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.244+0000 7f4534ff9700 1 -- 172.21.15.104:0/1247080681 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f452c012370 con 0x7f453809c090 2024-09-18T14:28:18.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.244+0000 7f4534ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:28:18.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.244+0000 7f4534ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:28:18.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.404+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 --> v2:172.21.15.174:6812/17603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45381219d0 con 0x7f4524035a80 2024-09-18T14:28:18.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.408+0000 7f4534ff9700 1 -- 172.21.15.104:0/1247080681 <== mgr.4114 v2:172.21.15.174:6812/17603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51958 (secure 0 0 0) 0x7f452800f620 con 0x7f4524035a80 2024-09-18T14:28:18.418 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-18T14:28:18.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 >> v2:172.21.15.174:6812/17603 conn(0x7f4524035a80 msgr2=0x7f4524037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 --2- 172.21.15.104:0/1247080681 >> v2:172.21.15.174:6812/17603 conn(0x7f4524035a80 0x7f4524037f30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f453811f9e0 tx=0x7f4528007ea0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 10 monclient: shutdown 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 msgr2=0x7f453811e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 0x7f453811e480 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f452c00e8d0 tx=0x7f452c008040 comp rx=0 tx=0).stop 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 shutdown_connections 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 --2- 172.21.15.104:0/1247080681 >> v2:172.21.15.174:6812/17603 conn(0x7f4524035a80 0x7f4524037f30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453809fe10 0x7f453811ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7f453809c900 0x7f453811e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 --2- 172.21.15.104:0/1247080681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f453809c090 0x7f453811e480 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 >> 172.21.15.104:0/1247080681 conn(0x7f45380a6080 msgr2=0x7f45380a3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 shutdown_connections 2024-09-18T14:28:18.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.412+0000 7f453de00700 1 -- 172.21.15.104:0/1247080681 wait complete. 2024-09-18T14:28:18.421 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-09-18T14:28:18.431 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-09-18T14:28:16.522741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4795856,"kb_used_data":557400424,"kb_used_omap":58342,"kb_used_meta":0,"kb_avail":557400424,"statfs":{"total":575688990720,"available":570778034176,"internally_reserved":0,"allocated":570778034176,"data_stored":570778034176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59742886,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":326,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:28:05.626406+0000","last_change":"2024-09-18T14:28:05.626349+0000","last_active":"2024-09-18T14:28:05.626406+0000","last_peered":"2024-09-18T14:28:05.626406+0000","last_clean":"2024-09-18T14:28:05.626406+0000","last_became_active":"2024-09-18T14:21:32.371287+0000","last_became_peered":"2024-09-18T14:21:32.371287+0000","last_unstale":"2024-09-18T14:28:05.626406+0000","last_undegraded":"2024-09-18T14:28:05.626406+0000","last_fullsized":"2024-09-18T14:28:05.626406+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:28:05.626207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:28:05.626207+0000","last_clean_scrub_stamp":"2024-09-18T14:28:05.626207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:05:12.609633+0000","scrub_duration":0.0031829699999999998,"objects_trimmed":0,"snaptrim_duration":0.00091715099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":317,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:27:57.620785+0000","last_change":"2024-09-18T14:27:57.620741+0000","last_active":"2024-09-18T14:27:57.620785+0000","last_peered":"2024-09-18T14:27:57.620785+0000","last_clean":"2024-09-18T14:27:57.620785+0000","last_became_active":"2024-09-18T14:21:32.281203+0000","last_became_peered":"2024-09-18T14:21:32.281203+0000","last_unstale":"2024-09-18T14:27:57.620785+0000","last_undegraded":"2024-09-18T14:27:57.620785+0000","last_fullsized":"2024-09-18T14:27:57.620785+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:27:57.620637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:27:57.620637+0000","last_clean_scrub_stamp":"2024-09-18T14:27:57.620637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:06:02.452270+0000","scrub_duration":0.0028727319999999998,"objects_trimmed":0,"snaptrim_duration":0.00079433699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":326,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:28:03.678139+0000","last_change":"2024-09-18T14:28:03.678081+0000","last_active":"2024-09-18T14:28:03.678139+0000","last_peered":"2024-09-18T14:28:03.678139+0000","last_clean":"2024-09-18T14:28:03.678139+0000","last_became_active":"2024-09-18T14:21:32.283225+0000","last_became_peered":"2024-09-18T14:21:32.283225+0000","last_unstale":"2024-09-18T14:28:03.678139+0000","last_undegraded":"2024-09-18T14:28:03.678139+0000","last_fullsized":"2024-09-18T14:28:03.678139+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:28:03.677929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:28:03.677929+0000","last_clean_scrub_stamp":"2024-09-18T14:28:03.677929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:54:21.087803+0000","scrub_duration":0.0035278670000000001,"objects_trimmed":0,"snaptrim_duration":0.00085728900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":317,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:27:58.666201+0000","last_change":"2024-09-18T14:27:58.666144+0000","last_active":"2024-09-18T14:27:58.666201+0000","last_peered":"2024-09-18T14:27:58.666201+0000","last_clean":"2024-09-18T14:27:58.666201+0000","last_became_active":"2024-09-18T14:21:32.279714+0000","last_became_peered":"2024-09-18T14:21:32.279714+0000","last_unstale":"2024-09-18T14:27:58.666201+0000","last_undegraded":"2024-09-18T14:27:58.666201+0000","last_fullsized":"2024-09-18T14:27:58.666201+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:27:58.666009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:27:58.666009+0000","last_clean_scrub_stamp":"2024-09-18T14:27:58.666009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:01:14.523270+0000","scrub_duration":0.0029856790000000002,"objects_trimmed":0,"snaptrim_duration":0.000717112,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"327'3","reported_seq":320,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:28:06.648671+0000","last_change":"2024-09-18T14:28:06.648603+0000","last_active":"2024-09-18T14:28:06.648671+0000","last_peered":"2024-09-18T14:28:06.648671+0000","last_clean":"2024-09-18T14:28:06.648671+0000","last_became_active":"2024-09-18T14:21:32.281362+0000","last_became_peered":"2024-09-18T14:21:32.281362+0000","last_unstale":"2024-09-18T14:28:06.648671+0000","last_undegraded":"2024-09-18T14:28:06.648671+0000","last_fullsized":"2024-09-18T14:28:06.648671+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'3","last_scrub_stamp":"2024-09-18T14:28:06.648461+0000","last_deep_scrub":"327'3","last_deep_scrub_stamp":"2024-09-18T14:28:06.648461+0000","last_clean_scrub_stamp":"2024-09-18T14:28:06.648461+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:56:31.722251+0000","scrub_duration":0.0058483820000000001,"objects_trimmed":0,"snaptrim_duration":0.00075824099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":315,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:27:59.992682+0000","last_change":"2024-09-18T14:27:59.992624+0000","last_active":"2024-09-18T14:27:59.992682+0000","last_peered":"2024-09-18T14:27:59.992682+0000","last_clean":"2024-09-18T14:27:59.992682+0000","last_became_active":"2024-09-18T14:21:32.280029+0000","last_became_peered":"2024-09-18T14:21:32.280029+0000","last_unstale":"2024-09-18T14:27:59.992682+0000","last_undegraded":"2024-09-18T14:27:59.992682+0000","last_fullsized":"2024-09-18T14:27:59.992682+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:27:59.992488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:27:59.992488+0000","last_clean_scrub_stamp":"2024-09-18T14:27:59.992488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:10:19.889160+0000","scrub_duration":0.0029142830000000002,"objects_trimmed":0,"snaptrim_duration":0.00065986400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":326,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:28:00.652483+0000","last_change":"2024-09-18T14:28:00.652381+0000","last_active":"2024-09-18T14:28:00.652483+0000","last_peered":"2024-09-18T14:28:00.652483+0000","last_clean":"2024-09-18T14:28:00.652483+0000","last_became_active":"2024-09-18T14:21:32.282531+0000","last_became_peered":"2024-09-18T14:21:32.282531+0000","last_unstale":"2024-09-18T14:28:00.652483+0000","last_undegraded":"2024-09-18T14:28:00.652483+0000","last_fullsized":"2024-09-18T14:28:00.652483+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:28:00.652220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:28:00.652220+0000","last_clean_scrub_stamp":"2024-09-18T14:28:00.652220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:50:02.303826+0000","scrub_duration":0.0035094700000000002,"objects_trimmed":0,"snaptrim_duration":0.0013003260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"327'2","reported_seq":319,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:28:05.679118+0000","last_change":"2024-09-18T14:28:05.679056+0000","last_active":"2024-09-18T14:28:05.679118+0000","last_peered":"2024-09-18T14:28:05.679118+0000","last_clean":"2024-09-18T14:28:05.679118+0000","last_became_active":"2024-09-18T14:21:32.280004+0000","last_became_peered":"2024-09-18T14:21:32.280004+0000","last_unstale":"2024-09-18T14:28:05.679118+0000","last_undegraded":"2024-09-18T14:28:05.679118+0000","last_fullsized":"2024-09-18T14:28:05.679118+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'2","last_scrub_stamp":"2024-09-18T14:28:05.678914+0000","last_deep_scrub":"327'2","last_deep_scrub_stamp":"2024-09-18T14:28:05.678914+0000","last_clean_scrub_stamp":"2024-09-18T14:28:05.678914+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:53:11.158287+0000","scrub_duration":0.00578708,"objects_trimmed":0,"snaptrim_duration":0.00081961700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"327'74","reported_seq":416,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-18T14:27:59.653590+0000","last_change":"2024-09-18T14:27:59.653490+0000","last_active":"2024-09-18T14:27:59.653590+0000","last_peered":"2024-09-18T14:27:59.653590+0000","last_clean":"2024-09-18T14:27:59.653590+0000","last_became_active":"2024-09-18T14:21:28.173928+0000","last_became_peered":"2024-09-18T14:21:28.173928+0000","last_unstale":"2024-09-18T14:27:59.653590+0000","last_undegraded":"2024-09-18T14:27:59.653590+0000","last_fullsized":"2024-09-18T14:27:59.653590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'74","last_scrub_stamp":"2024-09-18T14:27:59.653329+0000","last_deep_scrub":"327'74","last_deep_scrub_stamp":"2024-09-18T14:27:59.653329+0000","last_clean_scrub_stamp":"2024-09-18T14:27:59.653329+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:05.501503+0000","scrub_duration":0.0093523019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9395,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9620807,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.77400000000000002,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.90200000000000002,"5min":1.204,"15min":1.204},"last":0.746},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.55700000000000005,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.77700000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47899999999999998},"max":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91400000000000003,"15min":0.89600000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.38500000000000001},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":3.742},"last":0.91100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.84299999999999997,"15min":0.85899999999999999},"min":{"1min":0.63400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.6240000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.98299999999999998,"15min":0.89500000000000002},"min":{"1min":0.93400000000000005,"5min":0.623,"15min":0.432},"max":{"1min":1.361,"5min":1.927,"15min":1.927},"last":0.97899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.76400000000000001,"15min":0.79500000000000004},"min":{"1min":0.56799999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.2030000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.999,"15min":1.0309999999999999},"min":{"1min":0.74199999999999999,"5min":0.67700000000000005,"15min":0.57499999999999996},"max":{"1min":1.452,"5min":2.0590000000000002,"15min":4.0919999999999996},"last":0.69999999999999996}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76900000000000002,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.944,"5min":1.944,"15min":3.218},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.92600000000000005,"15min":1.006},"min":{"1min":0.72199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.5779999999999998},"last":0.94199999999999995}]}]},{"osd":4,"up_from":11,"seq":47244640340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799008,"kb_used_data":92900372,"kb_used_omap":9897,"kb_used_meta":0,"kb_avail":92900372,"statfs":{"total":95948165120,"available":95129980928,"internally_reserved":0,"allocated":95129980928,"data_stored":95129980928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10135067,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":1.3260000000000001,"5min":1.988,"15min":1.988},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.91000000000000003,"15min":0.97199999999999998},"min":{"1min":0.72199999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.111,"5min":2.012,"15min":2.012},"last":0.874}]},{"osd":1,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.79300000000000004},"min":{"1min":0.66400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0980000000000001,"5min":2.714,"15min":2.714},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89100000000000001,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93999999999999995,"5min":2.7650000000000001,"15min":2.7650000000000001},"last":0.70599999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.875,"15min":0.91500000000000004},"min":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.083,"5min":1.345,"15min":1.99},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.94699999999999995,"15min":1.0109999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.046,"5min":1.361,"15min":2.1240000000000001},"last":0.95399999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.69199999999999995,"15min":0.72799999999999998},"min":{"1min":0.56399999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.222},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.91400000000000003},"min":{"1min":0.60899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1990000000000001,"15min":1.7769999999999999},"last":0.66500000000000004}]},{"osd":5,"last update":"Wed Sep 18 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73099999999999998,"15min":0.88700000000000001},"min":{"1min":0.56299999999999994,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.012,"5min":1.2070000000000001,"15min":3.0609999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84499999999999997,"15min":0.99099999999999999},"min":{"1min":0.65200000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.075,"5min":1.29,"15min":3.2799999999999998},"last":0.98299999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640340,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":9537,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9766051,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81399999999999995,"15min":0.93000000000000005},"min":{"1min":0.56100000000000005,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0109999999999999,"5min":3.8359999999999999,"15min":5.9100000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.98299999999999998},"min":{"1min":0.52500000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0429999999999999,"5min":3.5609999999999999,"15min":6.173},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.89600000000000002,"15min":1.034},"min":{"1min":0.53300000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.131,"5min":4.2519999999999998,"15min":5.8789999999999996},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.95599999999999996,"15min":0.89400000000000002},"min":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3140000000000001,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":0.91800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90600000000000003,"15min":1.034},"min":{"1min":0.70399999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2250000000000001,"5min":5.9530000000000003,"15min":6.3109999999999999},"last":0.995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.95999999999999996,"15min":1.0649999999999999},"min":{"1min":0.71899999999999997,"5min":0.499,"15min":0.499},"max":{"1min":1.0880000000000001,"5min":5.6159999999999997,"15min":6.2350000000000003},"last":0.83299999999999996}]},{"osd":4,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.095,"5min":2.2109999999999999,"15min":3.8420000000000001},"last":0.747},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79800000000000004,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.99399999999999999,"5min":2.4169999999999998,"15min":5.8410000000000002},"last":0.88300000000000001}]},{"osd":5,"last update":"Wed Sep 18 14:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.81599999999999995,"15min":0.93700000000000006},"min":{"1min":0.61199999999999999,"5min":0.53700000000000003,"15min":0.52100000000000002},"max":{"1min":1.167,"5min":1.581,"15min":4.0739999999999998},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92500000000000004,"15min":1.01},"min":{"1min":0.628,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.397,"5min":2.6899999999999999,"15min":4.0940000000000003},"last":1.046}]}]},{"osd":2,"up_from":11,"seq":47244640340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799136,"kb_used_data":92900244,"kb_used_omap":9739,"kb_used_meta":0,"kb_avail":92900244,"statfs":{"total":95948165120,"available":95129849856,"internally_reserved":0,"allocated":95129849856,"data_stored":95129849856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9973025,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59399999999999997,"15min":0.56899999999999995},"min":{"1min":0.46700000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.878,"5min":1.849,"15min":1.849},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.68600000000000005,"15min":0.59799999999999998},"min":{"1min":0.52900000000000003,"5min":0.377,"15min":0.376},"max":{"1min":1.375,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78000000000000003,"15min":0.70199999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.206,"5min":4.7640000000000002,"15min":4.7640000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.84999999999999998,"15min":0.77700000000000002},"min":{"1min":0.56100000000000005,"5min":0.308,"15min":0.308},"max":{"1min":0.92800000000000005,"5min":6.7939999999999996,"15min":6.7939999999999996},"last":0.57199999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92900000000000005,"15min":0.89200000000000002},"min":{"1min":0.70499999999999996,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2350000000000001,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.92200000000000004},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.2729999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.85399999999999998}]},{"osd":4,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.91100000000000003,"15min":1.1830000000000001},"min":{"1min":0.69499999999999995,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.2949999999999999,"5min":1.778,"15min":9.0830000000000002},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.95899999999999996,"15min":1.1359999999999999},"min":{"1min":0.75700000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.218,"5min":2.0089999999999999,"15min":7.1399999999999997},"last":0.76000000000000001}]},{"osd":5,"last update":"Wed Sep 18 14:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.96099999999999997,"15min":1.1850000000000001},"min":{"1min":0.79400000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.27,"5min":2.0859999999999999,"15min":9.4900000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.94699999999999995,"15min":1.155},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0369999999999999,"5min":2.2429999999999999,"15min":8.3420000000000005},"last":0.78700000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640340,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799140,"kb_used_data":92900240,"kb_used_omap":9700,"kb_used_meta":0,"kb_avail":92900240,"statfs":{"total":95948165120,"available":95129845760,"internally_reserved":0,"allocated":95129845760,"data_stored":95129845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9933627,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.44700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67900000000000005,"5min":1.508,"15min":1.508},"last":0.495},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.28599999999999998},"max":{"1min":0.81599999999999995,"5min":1.038,"15min":1.038},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.251},"max":{"1min":0.92200000000000004,"5min":2.278,"15min":2.278},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74399999999999999,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.57799999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.88900000000000001,"15min":1.022},"min":{"1min":0.66000000000000003,"5min":0.41499999999999998,"15min":0.34699999999999998},"max":{"1min":1.0109999999999999,"5min":2.8239999999999998,"15min":5.6699999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.88,"5min":0.95099999999999996,"15min":0.84899999999999998},"min":{"1min":0.67900000000000005,"5min":0.53500000000000003,"15min":0.32900000000000001},"max":{"1min":1.196,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.79800000000000004}]},{"osd":4,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88600000000000001,"15min":1.002},"min":{"1min":0.72199999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.079,"5min":1.45,"15min":3.7490000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.92900000000000005,"15min":0.86899999999999999},"min":{"1min":0.65300000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.018,"5min":6.141,"15min":6.141},"last":0.82599999999999996}]},{"osd":5,"last update":"Wed Sep 18 14:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.873,"15min":0.98999999999999999},"min":{"1min":0.70899999999999996,"5min":0.46800000000000003,"15min":0.40600000000000003},"max":{"1min":1.111,"5min":1.292,"15min":4.2229999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94199999999999995,"15min":0.91200000000000003},"min":{"1min":0.69899999999999995,"5min":0.46100000000000002,"15min":0.41799999999999998},"max":{"1min":1.198,"5min":1.496,"15min":1.496},"last":0.86199999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640341,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":799716,"kb_used_data":92899664,"kb_used_omap":10072,"kb_used_meta":0,"kb_avail":92899664,"statfs":{"total":95948165120,"available":95129255936,"internally_reserved":0,"allocated":95129255936,"data_stored":95129255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10314309,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31},"max":{"1min":0.753,"5min":1.298,"15min":1.298},"last":0.55300000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.627,"15min":0.62},"min":{"1min":0.40200000000000002,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.80600000000000005,"5min":1.48,"15min":1.48},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.71199999999999997,"15min":0.69199999999999995},"min":{"1min":0.51400000000000001,"5min":0.46700000000000003,"15min":0.313},"max":{"1min":0.93500000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.79100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.872,"15min":0.84499999999999997},"min":{"1min":0.65800000000000003,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1759999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.53100000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97499999999999998,"5min":1.212,"15min":1.212},"last":0.65900000000000003}]},{"osd":4,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95799999999999996,"15min":0.97699999999999998},"min":{"1min":0.76900000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2470000000000001,"5min":1.579,"15min":1.9470000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.877,"15min":0.98299999999999998},"min":{"1min":0.627,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":2.476},"last":0.74299999999999999}]},{"osd":5,"last update":"Wed Sep 18 14:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.91300000000000003,"15min":0.89000000000000001},"min":{"1min":0.69299999999999995,"5min":0.56399999999999995,"15min":0.56299999999999994},"max":{"1min":1.2989999999999999,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93200000000000005,"15min":0.99399999999999999},"min":{"1min":0.73099999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.1539999999999999,"5min":1.3009999999999999,"15min":1.7989999999999999},"last":0.93600000000000005}]}]}],"pool_statfs":[]}} 2024-09-18T14:28:18.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T14:28:18.542 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient: get_monmap_and_config 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient: build_initial_monmap 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient: monmap: 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:28:18.541227+0000 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:28:18.541227+0000 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a-legacy 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c-legacy 2024-09-18T14:28:18.543 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-b 2024-09-18T14:28:18.544 INFO:teuthology.orchestra.run.smithi104.stderr:3: v2:172.21.15.104:3300/0 mon.noname-a 2024-09-18T14:28:18.544 INFO:teuthology.orchestra.run.smithi104.stderr:4: v2:172.21.15.174:3300/0 mon.noname-c 2024-09-18T14:28:18.544 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.545 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 1 Processor -- start 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 1 -- start start 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient: init 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faec409a6b0 0x7faec4131be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient(hunting): picked mon.noname-c con 0x7faec409a6b0 addr v2:172.21.15.174:3300/0 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faec409aa80 addr v1:172.21.15.104:6789/0 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faec4136f10 addr v1:172.21.15.174:6789/0 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec409b6e0 con 0x7faec409aa80 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec40a1280 con 0x7faec4136f10 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.546 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec409b9c0 con 0x7faec409a6b0 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec3fff700 10 monclient(hunting): authenticate will time out at 1219.530782s 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec27fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faec409aa80 0x7faec4132120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36784/0 (socket says 172.21.15.104:36784) 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec27fc700 1 -- 172.21.15.104:0/3486388873 learned_addr learned my addr 172.21.15.104:0/3486388873 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 1 --2- 172.21.15.104:0/3486388873 >> v2:172.21.15.174:3300/0 conn(0x7faec409a6b0 0x7faec4131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): get_auth_request con 0x7faec409a6b0 auth_method 0 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:28:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient(hunting): handle_auth_done global_id 5917 payload 274 2024-09-18T14:28:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 1 -- 172.21.15.104:0/3486388873 >> v1:172.21.15.174:6789/0 conn(0x7faec4136f10 legacy=0x7faec41392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:28:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 1 -- 172.21.15.104:0/3486388873 >> v1:172.21.15.104:6789/0 conn(0x7faec409aa80 legacy=0x7faec4132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T14:28:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient: _finish_hunting 0 2024-09-18T14:28:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 1 monclient: found mon.noname-c 2024-09-18T14:28:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:28:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.174:3300/0 2024-09-18T14:28:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.536+0000 7faec2ffd700 1 -- 172.21.15.104:0/3486388873 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec4099e60 con 0x7faec409a6b0 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec2ffd700 1 -- 172.21.15.104:0/3486388873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec409a6b0 msgr2=0x7faec4131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec2ffd700 1 --2- 172.21.15.104:0/3486388873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec409a6b0 0x7faec4131be0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7faec40a1c90 tx=0x7faeac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57a8c38e32dbd335 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 1 -- 172.21.15.104:0/3486388873 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac00b9a0 con 0x7faec409a6b0 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: dump: 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.550 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: _finish_auth 0 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 5 monclient: authenticate success, global_id 5917 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 1 -- 172.21.15.104:0/3486388873 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac00bb00 con 0x7faec409a6b0 2024-09-18T14:28:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 10 monclient: get_monmap_and_config success 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 4 set_mon_vals no callback set 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 1 -- 172.21.15.104:0/3486388873 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac00bea0 con 0x7faec409a6b0 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec1ffb700 10 monclient: dump: 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.552 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 10 monclient: shutdown 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 1 -- 172.21.15.104:0/3486388873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec409a6b0 msgr2=0x7faec4131be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 1 --2- 172.21.15.104:0/3486388873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec409a6b0 0x7faec4131be0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7faec40a1c90 tx=0x7faeac0091a0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 1 -- 172.21.15.104:0/3486388873 shutdown_connections 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 1 --2- 172.21.15.104:0/3486388873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec409a6b0 0x7faec4131be0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.540+0000 7faec3fff700 1 -- 172.21.15.104:0/3486388873 >> 172.21.15.104:0/3486388873 conn(0x7faec40a53b0 msgr2=0x7faec40a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 -- 172.21.15.104:0/3486388873 shutdown_connections 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 -- 172.21.15.104:0/3486388873 wait complete. 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient: build_initial_monmap 2024-09-18T14:28:18.553 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient: monmap: 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.noname-a 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.noname-b 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.noname-c 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 Processor -- start 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 -- start start 2024-09-18T14:28:18.554 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient: init 2024-09-18T14:28:18.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient: _reopen_session rank -1 2024-09-18T14:28:18.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T14:28:18.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 --2- >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faec409a6b0 0x7faec406f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): picked mon.noname-c con 0x7faec409a6b0 addr [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 0x7faec406adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): picked mon.noname-a con 0x7faec409aa80 addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec4136f10 0x7faec406b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): picked mon.noname-b con 0x7faec4136f10 addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec406c610 con 0x7faec409aa80 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec409aee0 con 0x7faec4136f10 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 1 -- --> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec40a3d80 con 0x7faec409a6b0 2024-09-18T14:28:18.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): _renew_subs 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec3fff700 10 monclient(hunting): authenticate will time out at 1219.538047s 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 0x7faec406adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 0x7faec406adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50512/0 (socket says 172.21.15.104:50512) 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 -- 172.21.15.104:0/3464062745 learned_addr learned my addr 172.21.15.104:0/3464062745 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): get_auth_request con 0x7faec409aa80 auth_method 0 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faec409a6b0 0x7faec406f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:28:18.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 10 monclient(hunting): get_auth_request con 0x7faec409a6b0 auth_method 0 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec37fe700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec4136f10 0x7faec406b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient(hunting): handle_auth_done global_id 15267 payload 824 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 -- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faec409a6b0 msgr2=0x7faec406f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faec409a6b0 0x7faec406f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 -- 172.21.15.104:0/3464062745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec4136f10 msgr2=0x7faec406b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec4136f10 0x7faec406b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient: _finish_hunting 0 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 monclient: found mon.noname-a 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec27fc700 1 -- 172.21.15.104:0/3464062745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac009420 con 0x7faec409aa80 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec37fe700 10 monclient: get_auth_request con 0x7faec4136f10 auth_method 0 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.544+0000 7faec37fe700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec4136f10 0x7faec406b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec27fc700 1 -- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 msgr2=0x7faec406adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec27fc700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 0x7faec406adf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c5e0 tx=0x7faebc004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc005d40 con 0x7faec409aa80 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T14:28:18.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: dump: 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: _finish_auth 0 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 5 monclient: authenticate success, global_id 15267 2024-09-18T14:28:18.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc0070d0 con 0x7faec409aa80 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 10 monclient: _renew_subs 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec406e880 con 0x7faec409aa80 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc007490 con 0x7faec409aa80 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec17fa700 4 set_mon_vals no callback set 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 10 monclient: dump: 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:fsid ad912f1a-f57b-4a10-a0fa-a3c1fef6a6d8 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-09-18T14:21:05.806012+0000 2024-09-18T14:28:18.564 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 17 (quincy) 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2: [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] mon.c 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 10 monclient: _renew_subs 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec406ef60 con 0x7faec409aa80 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 42763+0+0 (secure 0 0 0) 0x7faebc01b3b0 con 0x7faec409aa80 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec406e320 con 0x7faec409aa80 2024-09-18T14:28:18.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 1 --2- 172.21.15.104:0/3464062745 >> v2:172.21.15.174:6812/17603 conn(0x7faea4035a80 0x7faea4037f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T14:28:18.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(327..327 src has 124..327) v4 ==== 3502+0+0 (secure 0 0 0) 0x7faebc047410 con 0x7faec409aa80 2024-09-18T14:28:18.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec2ffd700 1 --2- 172.21.15.104:0/3464062745 >> v2:172.21.15.174:6812/17603 conn(0x7faea4035a80 0x7faea4037f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T14:28:18.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.548+0000 7faec2ffd700 10 monclient: get_auth_request con 0x7faea4035a80 auth_method 0 2024-09-18T14:28:18.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.552+0000 7faec2ffd700 1 --2- 172.21.15.104:0/3464062745 >> v2:172.21.15.174:6812/17603 conn(0x7faea4035a80 0x7faea4037f30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faeac0030c0 tx=0x7faeac016000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T14:28:18.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.556+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7faebc00f820 con 0x7faec409aa80 2024-09-18T14:28:18.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.556+0000 7faebaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T14:28:18.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.556+0000 7faebaffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T14:28:18.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.716+0000 7faec3fff700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T14:28:18.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.716+0000 7faec3fff700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T14:28:18.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.716+0000 7faec3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.104:3300/0 2024-09-18T14:28:18.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.716+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7faec4063340 con 0x7faec409aa80 2024-09-18T14:28:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.724+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7faebc00f820 con 0x7faec409aa80 2024-09-18T14:28:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.724+0000 7faebaffd700 10 monclient: handle_config config(1 keys) v1 2024-09-18T14:28:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.724+0000 7faec17fa700 4 set_mon_vals no callback set 2024-09-18T14:28:18.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.724+0000 7faec17fa700 10 set_mon_vals mon_health_to_clog = false 2024-09-18T14:28:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.724+0000 7faebaffd700 1 -- 172.21.15.104:0/3464062745 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7faebc017070 con 0x7faec409aa80 2024-09-18T14:28:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.724+0000 7faebaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T14:28:18.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.724+0000 7faebaffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 >> v2:172.21.15.174:6812/17603 conn(0x7faea4035a80 msgr2=0x7faea4037f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 --2- 172.21.15.104:0/3464062745 >> v2:172.21.15.174:6812/17603 conn(0x7faea4035a80 0x7faea4037f30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faeac0030c0 tx=0x7faeac016000 comp rx=0 tx=0).stop 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 10 monclient: shutdown 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 msgr2=0x7faec406adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 0x7faec406adf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c5e0 tx=0x7faebc004a80 comp rx=0 tx=0).stop 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 shutdown_connections 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 --2- 172.21.15.104:0/3464062745 >> v2:172.21.15.174:6812/17603 conn(0x7faea4035a80 0x7faea4037f30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.737 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faec4136f10 0x7faec406b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faec409aa80 0x7faec406adf0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 --2- 172.21.15.104:0/3464062745 >> [v2:172.21.15.104:3301/0,v1:172.21.15.104:6790/0] conn(0x7faec409a6b0 0x7faec406f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T14:28:18.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 >> 172.21.15.104:0/3464062745 conn(0x7faec40a53b0 msgr2=0x7faec409db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T14:28:18.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 shutdown_connections 2024-09-18T14:28:18.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-09-18T14:28:18.728+0000 7faec3fff700 1 -- 172.21.15.104:0/3464062745 wait complete. 2024-09-18T14:28:18.749 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T14:28:18.749 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T14:28:18.749 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T14:28:18.749 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:18.767 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T14:28:18.767 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T14:28:18.767 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:18.834 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T14:28:18.834 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T14:28:18.834 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:18.856 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T14:28:18.856 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T14:28:18.856 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:18.880 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T14:28:18.880 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-18T14:28:18.880 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:18.898 INFO:tasks.ceph.osd.4:Stopped 2024-09-18T14:28:18.898 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-18T14:28:18.898 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:18.956 INFO:tasks.ceph.osd.5:Stopped 2024-09-18T14:28:18.956 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T14:28:18.956 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T14:28:18.956 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:18.986 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T14:28:18.986 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T14:28:18.986 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T14:28:18.986 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:19.006 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T14:28:19.006 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T14:28:19.006 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:19.023 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T14:28:19.023 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T14:28:19.023 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:28:19.041 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T14:28:19.041 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T14:28:19.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-18T14:28:19.061 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:28:19.061 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T14:28:19.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:28:19.411 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T14:28:19.601 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:28:19.601 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T14:28:19.814 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:28:19.814 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-18T14:28:20.140 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:28:20.140 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-18T14:28:20.344 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:28:20.344 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-18T14:28:20.527 INFO:tasks.ceph:Archiving mon data... 2024-09-18T14:28:20.528 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/data/mon.a.tgz 2024-09-18T14:28:20.528 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-18T14:28:20.535 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.vNsFjtf4nu 2024-09-18T14:28:20.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.vNsFjtf4nu 2024-09-18T14:28:20.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.vNsFjtf4nu 2024-09-18T14:28:20.646 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.vNsFjtf4nu is 149KB 2024-09-18T14:28:20.664 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.vNsFjtf4nu 2024-09-18T14:28:20.671 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/data/mon.c.tgz 2024-09-18T14:28:20.671 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-18T14:28:20.719 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.9k9V75q2UL 2024-09-18T14:28:20.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9k9V75q2UL 2024-09-18T14:28:20.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.9k9V75q2UL 2024-09-18T14:28:20.790 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.9k9V75q2UL is 145KB 2024-09-18T14:28:20.805 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.9k9V75q2UL 2024-09-18T14:28:20.811 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/data/mon.b.tgz 2024-09-18T14:28:20.811 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-18T14:28:20.818 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.WIaJqxBmQa 2024-09-18T14:28:20.818 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WIaJqxBmQa 2024-09-18T14:28:20.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.WIaJqxBmQa 2024-09-18T14:28:20.936 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.WIaJqxBmQa is 151KB 2024-09-18T14:28:20.954 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.WIaJqxBmQa 2024-09-18T14:28:20.960 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T14:28:20.961 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T14:28:20.963 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T14:28:21.017 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T14:28:21.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T14:28:21.019 DEBUG:teuthology.orchestra.run.smithi174:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T14:28:21.042 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T14:28:21.045 INFO:teuthology.orchestra.run.smithi174.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T14:28:21.052 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T14:28:21.053 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/remote/smithi104/crash 2024-09-18T14:28:21.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T14:28:21.069 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/remote/smithi174/crash 2024-09-18T14:28:21.070 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T14:28:21.085 INFO:tasks.ceph:Compressing logs... 2024-09-18T14:28:21.085 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T14:28:21.114 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T14:28:21.142 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20169.log 2024-09-18T14:28:21.142 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20985.log 2024-09-18T14:28:21.143 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20360.log 2024-09-18T14:28:21.144 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20877.log 2024-09-18T14:28:21.144 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20985.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.20169.log.gz 2024-09-18T14:28:21.145 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19451.log 2024-09-18T14:28:21.145 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20360.log: /var/log/ceph/ceph-client.admin.20877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19025.log 2024-09-18T14:28:21.145 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.20985.log.gz 2024-09-18T14:28:21.145 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19367.log 2024-09-18T14:28:21.146 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.20360.log.gz 2024-09-18T14:28:21.146 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19451.log: /var/log/ceph/ceph-client.admin.19025.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.20877.log.gz 2024-09-18T14:28:21.146 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-18T14:28:21.147 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.19451.log.gz 2024-09-18T14:28:21.147 INFO:teuthology.orchestra.run.smithi104.stderr:gzip /var/log/ceph/ceph-client.admin.19367.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.18476.log 2024-09-18T14:28:21.147 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.19025.log.gz 2024-09-18T14:28:21.148 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.a.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.19367.log.gz 2024-09-18T14:28:21.149 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20029.log 2024-09-18T14:28:21.149 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19274.log 2024-09-18T14:28:21.150 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20633.log 2024-09-18T14:28:21.150 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.18476.log.gz 2024-09-18T14:28:21.150 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19340.log 2024-09-18T14:28:21.151 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19424.log 2024-09-18T14:28:21.151 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20633.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.20029.log.gz 2024-09-18T14:28:21.151 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18942.log 2024-09-18T14:28:21.151 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19340.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.19274.log.gz 2024-09-18T14:28:21.152 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19160.log 2024-09-18T14:28:21.152 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.19340.log.gz 2024-09-18T14:28:21.152 INFO:teuthology.orchestra.run.smithi104.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.20633.log.gz 2024-09-18T14:28:21.153 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18915.log 2024-09-18T14:28:21.154 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18942.log: /var/log/ceph/ceph-client.admin.19160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20387.log 2024-09-18T14:28:21.154 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.19424.log.gz 2024-09-18T14:28:21.154 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18915.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.18942.log.gz 2024-09-18T14:28:21.154 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.19160.log.gz-5 2024-09-18T14:28:21.154 INFO:teuthology.orchestra.run.smithi104.stderr:--verbose -- /var/log/ceph/ceph-client.admin.19642.log 2024-09-18T14:28:21.154 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.18915.log.gz 2024-09-18T14:28:21.154 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18305.log 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18658.log 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.20387.log.gz 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18626.log 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19642.log: /var/log/ceph/ceph-client.admin.18305.log: /var/log/ceph/ceph-client.admin.18658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18474.log 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.18305.log.gz 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19973.log 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18626.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.18658.log.gz 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18213.log 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.19642.log.gz 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.18626.log.gz 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.18474.log.gz 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.21012.log 2024-09-18T14:28:21.157 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20142.log 2024-09-18T14:28:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20169.log 2024-09-18T14:28:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20955.log 2024-09-18T14:28:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18674.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:gzip /var/log/ceph/ceph-client.admin.20955.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.22307.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.20169.log.gz 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21958.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18674.log: /var/log/ceph/ceph-client.admin.22307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18275.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.20955.log.gz 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17683.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21958.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.22307.log.gz 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21783.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18275.log: /var/log/ceph/ceph-client.admin.17683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19367.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21783.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.21958.log.gz 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.17683.log.gz 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.18674.log.gz 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21128.log 2024-09-18T14:28:21.164 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.21783.log.gz 2024-09-18T14:28:21.165 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19367.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.18275.log.gz 2024-09-18T14:28:21.165 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18639.log 2024-09-18T14:28:21.165 INFO:teuthology.orchestra.run.smithi174.stderr: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.19367.log.gz 2024-09-18T14:28:21.165 INFO:teuthology.orchestra.run.smithi174.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.21128.log: /var/log/ceph/ceph-client.admin.22133.log 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18471.log 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21012.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.19973.log.gz 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21228.log 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20142.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.18213.log.gz 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.21012.log.gz 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18861.log 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18471.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.20142.log.gz 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19478.log 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20306.log 2024-09-18T14:28:21.209 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.21228.log.gz 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18861.log: 89.2%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.18471.log.gz-- /var/log/ceph/ceph-client.admin.18834.log 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19397.log 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20306.log: /var/log/ceph/ceph-client.admin.18834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19079.log 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.18861.log.gz 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.19478.log.gz 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 88.3%--verbose -- replaced with /var/log/ceph/ceph-client.admin.20306.log.gz 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.19106.log 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19397.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.18834.log.gz 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20931.log 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.19397.log.gz 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18996.log 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19106.log: /var/log/ceph/ceph-client.admin.20931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20904.log 2024-09-18T14:28:21.210 INFO:teuthology.orchestra.run.smithi104.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.19079.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19669.log 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.20931.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.19106.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20904.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.18996.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17251.log 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19669.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.20904.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19753.log 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.19669.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.tmp-client.admin.17251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19534.log 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17251.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20552.log 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19753.log: /var/log/ceph/ceph-client.admin.19534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20660.log 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20552.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.19534.log.gz 2024-09-18T14:28:21.211 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18240.log 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.19753.log.gz 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18472.log 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.20552.log.gz 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:gzip /var/log/ceph/ceph-client.admin.18240.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19505.log 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18472.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.18240.log.gz 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.20660.log.gz 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18618.log 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.18472.log.gz 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.18618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18468.log 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.19505.log.gz 2024-09-18T14:28:21.212 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19052.log 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.18618.log.gz 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20198.log 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18780.log 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19052.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.18468.log.gz 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20688.log 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20198.log: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20742.log 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.19052.log.gz 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.20198.log.gz 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20115.log 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20688.log: gzip -5 --verbose -- 87.9%/var/log/ceph/ceph-client.admin.20002.log 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.18780.log.gz 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20742.log: /var/log/ceph/ceph-client.admin.20115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20850.log 89.5% 2024-09-18T14:28:21.213 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.20688.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.20742.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20002.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.20115.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.20002.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18969.log 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19944.log 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18313.log 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18969.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.20850.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19876.log 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21093.log 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.18969.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18313.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19944.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19876.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.18313.log.gz 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18314.log 2024-09-18T14:28:21.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20225.log 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.19876.log.gz 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.1.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.21093.log.gz 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18475.log 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18807.log 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20225.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17760.log 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.18314.log.gz/var/log/ceph/ceph-client.admin.18475.log: 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18807.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.20225.log.gz 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.18475.log.gz 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.18807.log.gz 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20252.log 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18311.log 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.17760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20252.log: /var/log/ceph/ceph-client.admin.20486.log 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.17760.log.gz 2024-09-18T14:28:21.215 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18311.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.20252.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.18311.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18082.log 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20056.log 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18306.log 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18082.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.20486.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20333.log 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.18082.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20279.log 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20796.log 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20333.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.20056.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20279.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.18306.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.20279.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.20333.log.gz 2024-09-18T14:28:21.216 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20769.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20606.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20796.log: /var/log/ceph/ceph-client.admin.20769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19242.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.20796.log.gz 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20606.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.20769.log.gz 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.20606.log.gz 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18167.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21174.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20088.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.19242.log.gz 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21147.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21174.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.18167.log.gz 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21201.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18888.log 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.21174.log.gz/var/log/ceph/ceph-client.admin.21147.log: 2024-09-18T14:28:21.217 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18312.log 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21201.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.20088.log.gz 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.21147.log.gz 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19615.log 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17640.log 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.21201.log.gz/var/log/ceph/ceph-client.admin.18312.log: 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21120.log 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19615.log: -- replaced with /var/log/ceph/ceph-client.admin.18888.log.gz 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20579.log 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.17640.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.18312.log.gz 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21120.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.19615.log.gz 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19588.log 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.21120.log.gz 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.17640.log.gz 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19724.log 2024-09-18T14:28:21.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20579.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19588.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.20579.log.gz 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18753.log 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19781.log 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.19588.log.gz 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.18753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20715.log 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.19724.log.gz 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19781.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.18753.log.gz 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.19781.log.gz 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21268.log 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19215.log 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21268.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.20715.log.gz 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr: 90.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T14:28:21.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19697.log 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.21268.log.gz 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21039.log 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19215.log: /var/log/ceph/ceph-client.admin.19697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18193.log 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21293.log 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr: 90.0% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.19697.log.gz/var/log/ceph/ceph-client.admin.21039.log: 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.19215.log.gz 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19561.log 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.21039.log.gz 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21293.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.18193.log.gz 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.21293.log.gz 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20958.log 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19561.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.19561.log.gz 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20823.log 2024-09-18T14:28:21.220 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21066.log 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.20823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18643.log 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.20958.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18625.log 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.20823.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18643.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.21066.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.18643.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19133.log 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18625.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.18625.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18141.log 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.19133.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.19133.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18267.log 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18141.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.18141.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.18267.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.18267.log.gz 2024-09-18T14:28:21.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19187.log 2024-09-18T14:28:21.222 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.19187.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.19187.log.gz 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18584.log 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21870.log 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.18639.log.gz 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21747.log 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.21128.log.gz 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21870.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.18584.log.gz 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.22133.log.gz 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20638.log 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21747.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.21870.log.gz 2024-09-18T14:28:21.238 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19094.log 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.21747.log.gz 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20029.log 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19130.log /var/log/ceph/ceph-client.admin.19094.log: 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.20638.log.gz 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19679.log 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19854.log 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.19094.log.gz 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19130.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.20029.log.gz 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19679.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22479.log 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.19130.log.gz 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mon.b.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.19679.log.gz 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20429.log 2024-09-18T14:28:21.239 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19854.log.gz 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21250.log 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21162.log 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.22479.log.gz 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18820.log 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21250.log: /var/log/ceph/ceph-client.admin.21162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20256.log 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18820.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.20429.log.gz 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.21250.log.gz 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.21162.log.gz 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.18820.log.gz 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18457.log 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22045.log 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20256.log: /var/log/ceph/ceph-client.admin.18457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19404.log 2024-09-18T14:28:21.240 INFO:teuthology.orchestra.run.smithi174.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.18457.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.20256.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20780.log 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22045.log: /var/log/ceph/ceph-client.admin.19404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19276.log 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19497.log 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21216.log 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19276.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.19404.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.22045.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19497.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.19276.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.20780.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.19497.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18547.log 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21781.log 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.21216.log.gz 2024-09-18T14:28:21.241 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19642.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18547.log: /var/log/ceph/ceph-client.admin.21781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18948.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18366.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18948.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.21781.log.gz 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19040.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19642.log.gz 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21608.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.18948.log.gz 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.18547.log.gz 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18366.log: /var/log/ceph/ceph-client.admin.19040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22097.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17724.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18911.log 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22097.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.18366.log.gz 2024-09-18T14:28:21.242 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.19040.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.21608.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.17724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21694.log 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22480.log 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.22097.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.17724.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18911.log: /var/log/ceph/ceph-client.admin.21694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21348.log 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21433.log 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22480.log: /var/log/ceph/ceph-client.admin.21348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20694.log 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21433.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.21694.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.21348.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20868.log 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.22480.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20694.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.18911.log.gz 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21302.log 2024-09-18T14:28:21.243 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22219.log 89.4% 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr: -- replaced with /var/log/ceph/ceph-client.admin.21433.log.gz 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.20694.log.gz 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20868.log: /var/log/ceph/ceph-client.admin.21302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22533.log 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22219.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.20868.log.gz 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.21302.log.gz 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.22219.log.gz 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20728.log 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21434.log 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20255.log 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20728.log: /var/log/ceph/ceph-client.admin.21434.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.22533.log.gz 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21076.log 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20728.log.gz 2024-09-18T14:28:21.244 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21921.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.21434.log.gz 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21251.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22446.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21921.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.21076.log.gz 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19995.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.20255.log.gz 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22009.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22567.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.21251.log.gz 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.21921.log.gz 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.22446.log.gz 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22009.log: /var/log/ceph/ceph-client.admin.20465.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22359.log 2024-09-18T14:28:21.245 INFO:teuthology.orchestra.run.smithi174.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.19995.log.gz 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18730.log 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20465.log: /var/log/ceph/ceph-client.admin.22359.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.22567.log.gz 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18765.log 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.22009.log.gz 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22393.log 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.22359.log.gz 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21782.log 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18765.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.20465.log.gz 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22393.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.18730.log.gz 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18947.log 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.22393.log.gz 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 /var/log/ceph/ceph-osd.5.log: --verbose -- /var/log/ceph/ceph-client.admin.18403.log 2024-09-18T14:28:21.246 INFO:teuthology.orchestra.run.smithi174.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.18765.log.gz 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.21782.log.gz 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18494.log 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21696.log 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20516.log 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18494.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.18947.log.gz 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21696.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.18403.log.gz 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.18494.log.gz 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.21696.log.gz 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19821.log 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20637.log 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:gzip /var/log/ceph/ceph-client.admin.19821.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19767.log 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18585.log 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20637.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.19821.log.gz 2024-09-18T14:28:21.247 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19767.log: gzip 89.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.20516.log.gz 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:--verbose -- /var/log/ceph/ceph-client.admin.19942.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18821.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20081.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19942.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.20637.log.gz 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18821.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.19767.log.gz 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.19942.log.gz 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.18585.log.gz 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.18821.log.gz 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20552.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18493.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20081.log: /var/log/ceph/ceph-client.admin.20552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20377.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18493.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22566.log 2024-09-18T14:28:21.248 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.18493.log.gz 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.20552.log.gz 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mgr.x.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.20081.log.gz 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.20377.log.gz 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19038.log 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20464.log 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20604.log 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19038.log: /var/log/ceph/ceph-client.admin.20464.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.20464.log.gz 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.22566.log.gz 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.19038.log.gz 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20463.log 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20378.log 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20604.log: /var/log/ceph/ceph-client.admin.20463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19002.log 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr:gzip /var/log/ceph/ceph-client.admin.20378.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19457.log 2024-09-18T14:28:21.249 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.20604.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19002.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.20463.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21748.log 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.20378.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph.audit.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.19002.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19820.log 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.19457.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21748.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.21748.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22185.log 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19943.log 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19820.log: /var/log/ceph/ceph-client.admin.22185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21347.log 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21400.log 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19943.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.22185.log.gz 2024-09-18T14:28:21.250 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21347.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19820.log.gz 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21486.log 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.19943.log.gz 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21868.log 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.21347.log.gz/var/log/ceph/ceph-client.admin.21400.log: 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22481.log 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18729.log 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21868.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.21400.log.gz 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22481.log: /var/log/ceph/ceph-client.admin.20028.log 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21163.log 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18729.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.21486.log.gz 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.21868.log.gz/var/log/ceph/ceph-client.admin.20028.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.22481.log.gz 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:28:21.251 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21164.log 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19220.log 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.18729.log.gz 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21163.log: /var/log/ceph/ceph-client.admin.21164.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.20028.log.gz 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.21163.log.gz 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20517.log 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.21164.log.gz 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22098.log 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20168.log 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20115.log 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22098.log: /var/log/ceph/ceph-client.admin.20168.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20517.log.gz 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19220.log.gz 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19551.log 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.22098.log.gz 2024-09-18T14:28:21.252 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20115.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.20168.log.gz 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.20115.log.gz 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21303.log 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18858.log 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19551.log: /var/log/ceph/ceph-client.admin.21303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19768.log 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19994.log 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18858.log: -- replaced with /var/log/ceph/ceph-client.admin.19551.log.gz 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:gzip /var/log/ceph/ceph-client.admin.19768.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.21607.log 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.21303.log.gz 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19769.log 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.19768.log.gz 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.18858.log.gz 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21607.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19994.log.gz 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18676.log 2024-09-18T14:28:21.253 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.21607.log.gz--verbose 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:-- /var/log/ceph/ceph-client.admin.22271.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19769.log: /var/log/ceph/ceph-client.admin.18676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17812.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20289.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19641.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr: -- replaced with /var/log/ceph/ceph-client.admin.19769.log.gz/var/log/ceph/ceph-client.admin.17812.log: 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.18676.log.gz 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20289.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.17812.log.gz 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22220.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21660.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.22271.log.gz 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19641.log: /var/log/ceph/ceph-client.admin.22220.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.20289.log.gz 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22358.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 /var/log/ceph/ceph-client.admin.21660.log: --verbose -- /var/log/ceph/ceph-client.admin.21399.log 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.22220.log.gz 2024-09-18T14:28:21.254 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22358.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.19641.log.gz 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.21660.log.gz 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22532.log 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.22358.log.gz 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20290.log 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19129.log 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22532.log: /var/log/ceph/ceph-client.admin.20290.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20290.log.gz 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.21399.log.gz 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22131.log 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.22532.log.gz 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20867.log 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19129.log: /var/log/ceph/ceph-client.admin.22131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17658.log 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20291.log 2024-09-18T14:28:21.255 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.17658.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19129.log.gz 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.22131.log.gz 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19311.log 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.17658.log.gz-5 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:--verbose -- /var/log/ceph/ceph-client.admin.22184.log 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20291.log: /var/log/ceph/ceph-client.admin.19311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18583.log 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.20867.log.gz 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22184.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.20291.log.gz 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19311.log.gz 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19907.log 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.22184.log.gz 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20781.log 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:gzip /var/log/ceph/ceph-client.admin.19907.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.20116.log 2024-09-18T14:28:21.256 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20988.log 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20781.log: /var/log/ceph/ceph-client.admin.20116.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.18583.log.gz 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20781.log.gz 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19907.log.gz 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21573.log 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.20116.log.gz 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22218.log 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20902.log 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19131.log 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22218.log: /var/log/ceph/ceph-client.admin.20902.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.20988.log.gz 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20030.log 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.21573.log.gz 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19131.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.20902.log.gz 2024-09-18T14:28:21.257 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.22218.log.gz 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.19131.log.gz 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18311.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20203.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17751.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18311.log: /var/log/ceph/ceph-client.admin.20203.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.20030.log.gz 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22306.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.18311.log.gz 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.17751.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.20203.log.gz 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.17751.log.gz 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18365.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18912.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22306.log: /var/log/ceph/ceph-client.admin.18365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22043.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19586.log 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.22306.log.gz 2024-09-18T14:28:21.258 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21435.log 2024-09-18T14:28:21.259 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22043.log: 89.2% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.18365.log.gz -- replaced with /var/log/ceph/ceph-client.admin.18912.log.gz 2024-09-18T14:28:21.259 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:28:21.259 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19586.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.22043.log.gz 2024-09-18T14:28:21.259 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19586.log.gz 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21835.log 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21521.log 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18857.log 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22394.log 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21521.log: /var/log/ceph/ceph-client.admin.18857.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.21435.log.gz 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.21835.log.gz 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17776.log 2024-09-18T14:28:21.281 INFO:teuthology.orchestra.run.smithi174.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.21521.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.18857.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18767.log 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21661.log 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.17776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20082.log 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18767.log: /var/log/ceph/ceph-client.admin.21661.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.17776.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18675.log 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.22394.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:gzip /var/log/ceph/ceph-client.admin.20082.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19550.log 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.18767.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.21661.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18675.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.20082.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.18675.log.gz 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19458.log 2024-09-18T14:28:21.282 INFO:teuthology.orchestra.run.smithi174.stderr:gzip /var/log/ceph/ceph-client.admin.19550.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.20342.log 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18638.log 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19458.log: /var/log/ceph/ceph-client.admin.20342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19733.log 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18949.log 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4%/var/log/ceph/ceph-client.admin.18638.log: -- replaced with /var/log/ceph/ceph-client.admin.19458.log.gz 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3%/var/log/ceph/ceph-client.admin.19733.log: -- replaced with /var/log/ceph/ceph-client.admin.19550.log.gz 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19093.log 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.20342.log.gz 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.19733.log.gz 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18949.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.18638.log.gz 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20550.log 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.18949.log.gz 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19093.log: /var/log/ceph/ceph-client.admin.21077.log 2024-09-18T14:28:21.283 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20117.log 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20989.log 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21077.log: /var/log/ceph/ceph-client.admin.20117.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19093.log.gz 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.20550.log.gz 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.21077.log.gz 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19677.log 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.20117.log.gz 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21520.log /var/log/ceph/ceph-client.admin.20989.log: 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22044.log 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19677.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.20989.log.gz 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19402.log 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19908.log 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21075.log 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19402.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.22044.log.gz 89.0% -- replaced with /var/log/ceph/ceph-client.admin.21520.log.gz 2024-09-18T14:28:21.284 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19908.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.19677.log.gz 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.19908.log.gz 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19402.log.gz 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18312.log 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20551.log 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19312.log 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18312.log: /var/log/ceph/ceph-client.admin.20551.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.21075.log.gz 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20551.log.gz 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.18312.log.gz 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20901.log 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19312.log: /var/log/ceph/ceph-client.admin.20954.log 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22132.log 2024-09-18T14:28:21.285 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20901.log: gzip /var/log/ceph/ceph-client.admin.20954.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19366.log 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.19312.log.gz 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21522.log 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20693.log 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19366.log: 88.4% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.20954.log.gz 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr: -- replaced with /var/log/ceph/ceph-client.admin.22132.log.gz 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21522.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.20901.log.gz 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19366.log.gz 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.21522.log.gz 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18492.log 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18548.log 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22392.log 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18492.log: gzip /var/log/ceph/ceph-client.admin.18548.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19496.log 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22392.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.20693.log.gz 2024-09-18T14:28:21.286 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.18548.log.gz 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.18492.log.gz 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20903.log 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19588.log 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19496.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.22392.log.gz-5 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:--verbose -- /var/log/ceph/ceph-client.admin.21042.log 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21487.log 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.19496.log.gz 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19588.log: /var/log/ceph/ceph-client.admin.21042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21922.log 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21487.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.20903.log.gz 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21922.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.21042.log.gz 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.19588.log.gz 2024-09-18T14:28:21.287 INFO:teuthology.orchestra.run.smithi174.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.21487.log.gz 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19941.log 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.21922.log.gz 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20990.log 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21834.log 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18402.log 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20990.log: gzip /var/log/ceph/ceph-client.admin.21834.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.20727.log 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18402.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.20990.log.gz 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19941.log.gz 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.18402.log.gz 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.21834.log.gz 2024-09-18T14:28:21.288 INFO:teuthology.orchestra.run.smithi174.stderr: 85.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19855.log 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22305.log 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20727.log: /var/log/ceph/ceph-client.admin.19855.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.19855.log.gz 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.20727.log.gz 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18310.log 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20202.log 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22010.log 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18310.log: gzip /var/log/ceph/ceph-client.admin.20202.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.18401.log 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20376.log 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.22305.log.gz 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18401.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.18310.log.gz 2024-09-18T14:28:21.289 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21957.log 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.22010.log.gz 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.20202.log.gz 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19403.log 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21215.log 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21957.log: /var/log/ceph/ceph-client.admin.19403.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.18401.log.gz 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.21957.log.gz 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.19403.log.gz 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.20376.log.gz 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20430.log 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21215.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19039.log 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19221.log 2024-09-18T14:28:21.290 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20815.log 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19039.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.20430.log.gz 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19221.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.21215.log.gz 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18766.log 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.19039.log.gz 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20816.log 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.19221.log.gz 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20815.log: /var/log/ceph/ceph-client.admin.18766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19732.log 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22445.log 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.20815.log.gz 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20816.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.18766.log.gz 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19184.log 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21695.log 2024-09-18T14:28:21.291 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22445.log: /var/log/ceph/ceph-client.admin.19184.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.20816.log.gz 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.19732.log.gz 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.22445.log.gz 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19495.log 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.19184.log.gz 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20639.log 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21249.log 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19495.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.21695.log.gzgzip -5 /var/log/ceph/ceph-client.admin.20639.log: --verbose -- /var/log/ceph/ceph-client.admin.20729.log 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21249.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.19495.log.gz 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.20639.log.gz 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19856.log 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21956.log 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.21249.log.gz 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19856.log: 90.2%gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-18T14:28:21.292 INFO:teuthology.orchestra.run.smithi174.stderr: -- replaced with /var/log/ceph/ceph-client.admin.20729.log.gz 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19313.log 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.19856.log.gz 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21574.log 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18856.log 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19313.log: /var/log/ceph/ceph-client.admin.21574.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.21956.log.gz 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.19313.log.gz 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.21574.log.gz 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18276.log 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21609.log 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18856.log: /var/log/ceph/ceph-client.admin.18276.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.18276.log.gz 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.18856.log.gz 2024-09-18T14:28:21.293 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21869.log 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21129.log /var/log/ceph/ceph-client.admin.21609.log: 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22568.log 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21129.log: 90.4% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.21869.log.gz 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr: -- replaced with /var/log/ceph/ceph-client.admin.21609.log.gz 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.21129.log.gz 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21041.log 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.22568.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.22568.log.gz 2024-09-18T14:28:21.299 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22272.log 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19587.log 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.21041.log: /var/log/ceph/ceph-client.admin.22272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19275.log 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19587.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.22272.log.gz 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.21041.log.gz 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19678.log 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.19587.log.gz 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20204.log 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19275.log: gzip -5 /var/log/ceph/ceph-client.admin.19678.log: --verbose -- /var/log/ceph/ceph-client.admin.19003.log 2024-09-18T14:28:21.300 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20204.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.19678.log.gz 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20603.log 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.19275.log.gz 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19003.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.20204.log.gz 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19222.log 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.19003.log.gz 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19185.log 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19222.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.20603.log.gz 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.19222.log.gz 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20814.log 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.19185.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.19185.log.gz 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21346.log 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20343.log 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20814.log: /var/log/ceph/ceph-client.admin.21346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18456.log 2024-09-18T14:28:21.301 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.20343.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.20814.log.gz 2024-09-18T14:28:21.302 INFO:teuthology.orchestra.run.smithi174.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.20343.log.gz 2024-09-18T14:28:21.302 INFO:teuthology.orchestra.run.smithi174.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.21346.log.gz 2024-09-18T14:28:21.302 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.18456.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.18456.log.gz 2024-09-18T14:28:21.302 INFO:teuthology.orchestra.run.smithi174.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-18T14:28:21.339 INFO:teuthology.orchestra.run.smithi104.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-18T14:28:21.482 INFO:teuthology.orchestra.run.smithi104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-18T14:28:21.565 INFO:teuthology.orchestra.run.smithi174.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-18T14:28:22.355 INFO:teuthology.orchestra.run.smithi104.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-18T14:28:22.360 INFO:teuthology.orchestra.run.smithi174.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-18T14:28:22.412 INFO:teuthology.orchestra.run.smithi104.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-18T14:28:22.421 INFO:teuthology.orchestra.run.smithi104.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-18T14:28:22.422 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-18T14:28:22.423 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m1.306s 2024-09-18T14:28:22.423 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m4.109s 2024-09-18T14:28:22.423 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.236s 2024-09-18T14:28:22.429 INFO:teuthology.orchestra.run.smithi174.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-18T14:28:22.484 INFO:teuthology.orchestra.run.smithi174.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-18T14:28:22.485 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-18T14:28:22.485 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m1.354s 2024-09-18T14:28:22.485 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m4.224s 2024-09-18T14:28:22.485 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.250s 2024-09-18T14:28:22.486 INFO:tasks.ceph:Archiving logs... 2024-09-18T14:28:22.486 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/remote/smithi104/log 2024-09-18T14:28:22.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T14:28:23.175 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/remote/smithi174/log 2024-09-18T14:28:23.176 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T14:28:23.967 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T14:28:23.976 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T14:28:23.977 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T14:28:23.979 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T14:28:24.032 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T14:28:24.040 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T14:28:24.040 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T14:28:24.043 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T14:28:24.064 INFO:teuthology.orchestra.run.smithi104.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T14:28:24.064 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================== 2024-09-18T14:28:24.064 INFO:teuthology.orchestra.run.smithi104.stdout:*hv01.front.sepi 67.205.162.81 3 u 30 64 377 0.103 0.604 1.296 2024-09-18T14:28:24.064 INFO:teuthology.orchestra.run.smithi104.stdout:+hv02.front.sepi 96.245.170.99 3 u 25 64 377 0.074 5.657 2.534 2024-09-18T14:28:24.064 INFO:teuthology.orchestra.run.smithi104.stdout:+hv03.front.sepi 74.6.168.72 3 u 24 64 377 0.068 0.755 1.387 2024-09-18T14:28:24.064 INFO:teuthology.orchestra.run.smithi104.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:28:24.066 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T14:28:24.066 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-09-18T14:28:24.066 INFO:teuthology.orchestra.run.smithi174.stdout:*hv01.front.sepi 67.205.162.81 3 u 31 64 377 0.083 1.874 1.640 2024-09-18T14:28:24.066 INFO:teuthology.orchestra.run.smithi174.stdout:+hv02.front.sepi 96.245.170.99 3 u 28 64 377 0.069 4.642 1.649 2024-09-18T14:28:24.066 INFO:teuthology.orchestra.run.smithi174.stdout:+hv03.front.sepi 74.6.168.72 3 u 25 64 377 0.093 1.725 2.061 2024-09-18T14:28:24.066 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:28:24.067 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T14:28:24.076 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T14:28:24.076 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T14:28:24.085 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T14:28:24.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T14:28:24.102 INFO:teuthology.task.internal:Duration was 897.903739 seconds 2024-09-18T14:28:24.102 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T14:28:24.113 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T14:28:24.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T14:28:24.115 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T14:28:24.168 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T14:28:24.168 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-09-18T14:28:24.168 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T14:28:24.180 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T14:28:24.181 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T14:28:24.194 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T14:28:24.194 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T14:28:24.196 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T14:28:24.233 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T14:28:24.233 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T14:28:24.278 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T14:28:24.350 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T14:28:24.359 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T14:28:24.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T14:28:24.371 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T14:28:24.388 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T14:28:24.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:28:24.414 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:28:24.427 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-09-18T14:28:24.429 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-09-18T14:28:24.460 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:28:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:28:24.466 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:28:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:28:24.505 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T14:28:24.515 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T14:28:24.515 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/remote/smithi104 2024-09-18T14:28:24.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T14:28:24.542 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911101/remote/smithi174 2024-09-18T14:28:24.542 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T14:28:24.567 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T14:28:24.568 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T14:28:24.582 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T14:28:24.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T14:28:24.626 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T14:28:24.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T14:28:24.634 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T14:28:24.634 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T14:28:24.637 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T14:28:24.642 INFO:teuthology.orchestra.run.smithi104.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 14:28 /home/ubuntu/cephtest 2024-09-18T14:28:24.644 INFO:teuthology.orchestra.run.smithi174.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 14:28 /home/ubuntu/cephtest 2024-09-18T14:28:24.647 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T14:28:24.656 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T14:28:24.677 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/pool-create-delete} duration: 897.9037394523621 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T14:28:24.678 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T14:28:24.753 INFO:teuthology.run:pass